Encoding apparatus and encoding method as well as decoding apparatus and decoding method

ABSTRACT

The encoding apparatus encodes an input image by a non-reversible encoding method and transmits identification information for identifying match components that are, from among a plurality of texture components registered in a database, texture components that match with the input image and encoded data obtained by encoding the input image. The decoding apparatus receives the encoded data and the identification information, decodes the encoded data into a decoded image, and synthesizes the texture component as the match component identified by the identification information from among a plurality of texture components registered in a database and the decoded image. The present technology can be applied, for example, a codec or the like by which an image is encoded and decoded.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation application of U.S. patentapplication Ser. No. 16/094,084, filed Oct. 16, 2018 which is a U.S.National Phase of International Patent Application No. PCT/JP2017/014454filed Apr. 7, 2017, which claims priority benefit of Japanese PatentApplication No. JP 2016-086215 filed Apr. 22, 2016 in the Japan PatentOffice. Each of the above-referenced applications is hereby incorporatedherein by reference in its entirety.

TECHNICAL FIELD

The present technology relates to an encoding apparatus and an encodingmethod as well as a decoding apparatus and a decoding method, andparticularly to an encoding apparatus and an encoding method as well asa decoding apparatus and a decoding method by which, for example, atransmission efficiency and picture quality can be improved.

BACKGROUND ART

In an image, a texture is, for example, fine patterns or the like in animage and frequently is a signal of a high frequency (high frequencyband). For example, in an encoding method based on FFT (Fast FourierTransform) (frequency band resolution) such as AVC (Advanced VideoCoding), an image is degraded mainly from high frequency components, andtherefore, (part of) a texture of the image is lost and the picturequality of an image obtained on the decoding side is degraded.

As a method for suppressing degradation of the picture quality caused byloss of a texture of an image, a method is available by which, in orderto restore a texture lost upon encoding of an image, a texture of animage is transmitted separately in addition to encoded data of anencoded image (for example, refer to PTL 1 or NPL 1).

For example, in the method based on NPL 1, texture components areremoved from an original image in an encoding apparatus and the originalimage after the texture components are removed is encoded. Then, in theencoding apparatus, a small amount of texture components (image) andsynthesis parameters are transmitted together with encoded data obtainedby encoding.

On the other hand, in a decoding apparatus, the encoded data is decodedand the texture components from the encoding apparatus are synthesizedwith a decoded image obtained by the decoding using the synthesisparameters and then a restoration image obtained by restoring theoriginal image is generated.

In PTL 1, a technology is proposed in which, in an encoding apparatus, atexture (texture pattern) is reduced and transmitted and, in a decodingapparatus, the reduced texture is enlarged by a super-resolutiontechnology.

CITATION LIST Patent Literature

-   [PTL 1] WO 2011/090798-   [NPL 2] A. Dumitras and B. Haskell: “An Encoder-Decoder Texture    Replacement Method with Application to Content-Based Movie Coding,”    IEEE Transaction son Circuits and Systems for Video Technology, vol.    14, No. 6, June 2004

SUMMARY Technical Problem

In the case where the number of texture components (number of patterns)transmitted from the encoding apparatus to the decoding apparatus issmall, the texture of an input image cannot be restored sufficiently andthe picture quality of the restoration image is degraded.

Although, if the number of texture components to be transmitted from theencoding apparatus to the decoding apparatus is increased, then thepicture quality of the restoration image can be improved. However, thetransmission efficiency degrades as the number of texture componentstransmitted from the encoding apparatus to the decoding apparatusincreases.

It is to be noted that, although a method is available by which an imagehaving a fixed pattern or random noise or the like is deformed torestore a texture, according to such a method as just described, it isdifficult to generate a pattern like a thin line and a sufficient numberof textures cannot be represented.

Further, by an indeterminate problem that, in the case where a reducedtexture is enlarged by a super-resolution technology, it is difficult torestore high frequency components of a texture lost upon reduction, theexpressiveness of the texture degrades in comparison with that in thecase where a texture is transmitted without reduction.

The present technology has been made in view of such a situation asdescribed above, and it is an object of the present disclosure toimprove the transmission efficiency and picture quality.

In the case where the number of texture components (number of patterns)transmitted from the encoding apparatus to the decoding apparatus issmall, the texture of an input image cannot be restored sufficiently andthe picture quality of the restoration image is degraded.

Although, if the number of texture components to be transmitted from theencoding apparatus to the decoding apparatus is increased, then thepicture quality of the restoration image can be improved. However, thetransmission efficiency degrades as the number of texture componentstransmitted from the encoding apparatus to the decoding apparatusincreases.

It is to be noted that, although a method is available by which an imagehaving a fixed pattern or random noise or the like is deformed torestore a texture, according to such a method as just described, it isdifficult to generate a pattern like a thin line and a sufficient numberof textures cannot be represented.

Further, by an indeterminate problem that, in the case where a reducedtexture is enlarged by a super-resolution technology, it is difficult torestore high frequency components of a texture lost upon reduction, theexpressiveness of the texture degrades in comparison with that in thecase where a texture is transmitted without reduction.

The present technology has been made in view of such a situation asdescribed above, and it is an object of the present disclosure toimprove the transmission efficiency and picture quality.

Solution to Problems

The encoding apparatus of the present technology is an encodingapparatus including an encoding unit configured to encode an input imageby a non-reversible encoding method, a database in which a plurality oftexture components are registered, and a transmission unit configured totransmit identification information for identifying match componentsthat are, from among the plurality of texture components registered inthe database, the texture components that match with the input image andencoded data obtained by encoding the input image.

The encoding method of the present technology is an encoding methodincluding encoding an input image by a non-reversible encoding method,and transmitting identification information for identifying a matchcomponent that is a texture component that matches with the input imagefrom among a plurality of texture components registered in a database inwhich the plurality of texture components are registered and encodeddata obtained by encoding the input image.

In the encoding apparatus and the encoding method of the presenttechnology, an input image is encoded by a non-reversible encodingmethod, and identification information for identifying a match componentthat is a texture component that matches with the input image from amonga plurality of texture components registered in a database in which theplurality of texture components are registered and encoded data obtainedby encoding the input image are transmitted.

The decoding apparatus of the present technology is a decoding apparatusincluding a reception unit configured to receive encoded data obtainedby encoding an input image by a non-reversible encoding method andidentification information for identifying a match component that is atexture component that matches with the input image, a decoding unitconfigured to decode the encoded data into a decoded image, a databasein which a plurality of texture components are registered, and asynthesis unit configured to synthesize the texture component as thematch component identified by the identification information from amongthe plurality of texture components registered in the database and thedecoded image.

The decoding method of the present technology is a decoding methodincluding receiving encoded data obtained by encoding an input image bya non-reversible encoding method and identification information foridentifying a match component that is a texture component that matcheswith the input image, decoding the encoded data into a decoded image,and synthesizing the texture component as the match component identifiedby the identification information from among a plurality of texturecomponents registered in a database in which the plurality of texturecomponents are registered and the decoded image.

In the decoding apparatus and the decoding method of the presenttechnology, encoded data obtained by encoding an input image by anon-reversible encoding method and identification information foridentifying a match component that is a texture component that matcheswith the input image are received. Then, the encoded data is decodedinto a decoded image, and the texture component as the match componentidentified by the identification information from among a plurality oftexture components registered in a database in which the plurality oftexture components are registered and the decoded image are synthesized.

It is to be noted that the encoding apparatus and the decoding apparatuscan be implemented by causing a computer to execute a program.

Further, the program to be executed by a computer can be provided bytransmitting the same through a transmission medium or by recording thesame on a recording medium.

Furthermore, the encoding apparatus and the decoding apparatus may eachbe an independent apparatus or an internal block that configures oneapparatus.

Advantageous Effect of Invention

With the present technology, the transmission efficiency and picturequality can be improved.

It is to be noted that the advantageous effect described herein is notnecessarily restrictive, and any advantageous effect described in thepresent disclosure may be applicable.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram depicting a first example of a configurationof a codec that restores a texture lost upon encoding.

FIG. 2 is a block diagram depicting a second example of a configurationof a codec that restores a texture lost upon encoding.

FIG. 3 is a flow chart illustrating an example of an encoding process ofan encoding apparatus 30.

FIG. 4 is a flow chart illustrating an example of a decoding process ofa decoding apparatus 40.

FIG. 5 is a block diagram depicting a third example of a configurationof a codec that restores a texture lost upon encoding.

FIG. 6 is a block diagram depicting a fourth example of a configurationexample of a codec that restores a texture lost upon encoding.

FIG. 7 is a view illustrating an overview of basis learning forconverting texture components into a basis.

FIG. 8 is a view illustrating an overview of basis synthesis forgenerating texture components using a basis.

FIG. 9 is a flow chart illustrating an example of an encoding process ofan encoding apparatus 50.

FIG. 10 is a flow chart illustrating an example of a decoding process ofa decoding apparatus 60.

FIG. 11 is a block diagram depicting a fifth example of a configurationof a codec that restores a texture lost upon encoding.

FIG. 12 is a block diagram depicting a sixth example of a configurationof a codec that restores a texture lost upon encoding.

FIG. 13 is a flow chart illustrating an example of an encoding processof the encoding apparatus 50.

FIG. 14 is a flow chart illustrating an example of a decoding process ofthe decoding apparatus 60.

FIG. 15 is a block diagram depicting a seventh example of aconfiguration of a codec that restores a texture lost upon encoding.

FIG. 16 is a flow chart illustrating an example of a process performedby the encoding apparatus 50 and the decoding apparatus 60 when DB dataof a texture DB 63 is updated.

FIG. 17 is a block diagram depicting an eighth example of aconfiguration of a codec that restores a texture lost upon encoding.

FIG. 18 is a block diagram depicting a ninth example of a configurationof a codec that restores a texture lost upon encoding.

FIG. 19 is a block diagram depicting a tenth example of a configurationof a codec that restores a texture lost upon encoding.

FIG. 20 is a block diagram depicting an example of a configuration of aregistration unit 151.

FIG. 21 is a flow chart illustrating an example of an encoding processof the encoding apparatus 50.

FIG. 22 is a view depicting an example of a multi-view image encodingmethod.

FIG. 23 is a view depicting an example of a principal configuration of amulti-view image encoding apparatus to which the present technology isapplied.

FIG. 24 is a view depicting an example of a principal configuration ofthe multi-view image decoding apparatus to which the present technologyis applied.

FIG. 25 is a view depicting an example of a hierarchical image encodingmethod.

FIG. 26 is a view depicting an example of a principal configuration of ahierarchical image encoding apparatus to which the present technology isapplied.

FIG. 27 is a view depicting an example of a principal configuration of ahierarchical image decoding apparatus to which the present technology isapplied.

FIG. 28 is a block diagram depicting an example of a principalconfiguration of a computer.

FIG. 29 is a block diagram depicting an example of a schematicconfiguration of a television apparatus.

FIG. 30 is a block diagram depicting an example of a schematicconfiguration of a portable telephone set.

FIG. 31 is a block diagram depicting an example of a schematicconfiguration of a recording and reproduction apparatus.

FIG. 32 is a block diagram depicting an example of a schematicconfiguration of an image pickup apparatus.

FIG. 33 is a block diagram depicting an example of a schematicconfiguration of a video set.

FIG. 34 is a block diagram depicting an example of a schematicconfiguration of a video processor.

FIG. 35 is a block diagram depicting another example of a schematicconfiguration of the video processor.

DESCRIPTION OF EMBODIMENTS First Example of Configuration of Codec

FIG. 1 is a block diagram depicting a first example of a configurationof a codec that restores of a texture lost upon encoding.

Referring to FIG. 1, the codec includes an encoding apparatus 10 and adecoding apparatus 20.

The encoding apparatus 10 includes a texture component extraction unit11, a removal unit 12 and an encoding unit 13.

An original image (moving picture, still picture) as an input imageinputted to the encoding apparatus 10 is supplied to the texturecomponent extraction unit 11.

The texture component extraction unit 11 extracts and supplies texturecomponents of an input image from the input image to the removal unit 12and transmits the texture components to the decoding apparatus 20.

Not only the texture components of the input image but also the inputimage is supplied from the texture component extraction unit 11 to theremoval unit 12.

The removal unit 12 arithmetically operates the difference between theinput image and a texture component of the input image from the texturecomponent extraction unit 11 to remove the texture component of theinput image from the input image and supplies the input image from whichthe texture component is removed as an encoding target image of a targetof encoding by the encoding unit 13 to the encoding unit 13.

Here, the encoding target image obtained by the removal unit 12 is animage obtained by removing a texture component from the input image,namely, an image from which a high frequency component is removed, andtherefore, can be regarded as a low frequency component of the inputimage.

The encoding unit 13 encodes the encoding target image from the removalunit 12 by a non-reversible (irreversible) encoding method such as, forexample, MPEG (Moving Picture Experts Group), AVC, HEVC (High EfficiencyVideo Coding) or some other method as a hybrid method that is acombination of predictive coding and orthogonal transformation, andtransmits encoded data obtained as a result of the encoding to thedecoding apparatus 20.

Here, in encoding of the non-reversible encoding method by the encodingunit 13, since an image is converted into a signal of the frequencydomain and the signal of the frequency domain is quantized as in AVC orthe like, a frequency component included in the image and having a lowlevel, namely, for example, a high frequency component such as a texturecomponent or the like is lost.

The decoding apparatus 20 includes a decoding unit 21, a texturecomponent restoration unit 22 and a synthesis unit 23.

The decoding unit 21 accepts encoded data transmitted from (the encodingunit 13 of) the encoding apparatus 10 by receiving the encoded data anddecodes the encoded data by a method corresponding to the encodingmethod of the encoding unit 13. The decoding unit 21 supplies a decodedimage obtained by the decoding of the encoded data to the synthesis unit23.

Here, the decoded image obtained by the decoding unit 21 corresponds toa low frequency component of an encoding target image, namely, of theinput image.

The texture component restoration unit 22 receives a texture componenttransmitted from (the texture component extraction unit 11 of) theencoding apparatus 10, performs necessary processing to restore thetexture component of the input image and supplies the restored texturecomponent to the synthesis unit 23.

The synthesis unit 23 synthesizes a low frequency component of the inputimage as the decoded image from the decoding unit 21 and a texturecomponent of the input image from the texture component restoration unit22 to generate a restoration image that restores the input image(original image), and outputs the restoration image as an output imageoutputted from the decoding apparatus 20.

In the codec of FIG. 1, since a texture component of an input image istransmitted from the encoding apparatus 10 to the decoding apparatus 20,the decoding apparatus 20 can obtain an output image in which thetexture of the input image is restored, namely, can improve the picturequality of the output image.

However, in the codec of FIG. 1, since a texture component of an inputimage is transmitted from the encoding apparatus 10 to the decodingapparatus 20, the transmission efficiency degrades.

Although the transmission efficiency can be improved by decreasing thenumber of texture components (pattern number) to be transmitted from theencoding apparatus 10 to the decoding apparatus 20, if the number oftexture components to be transmitted from the encoding apparatus 10 tothe decoding apparatus 20 is decreased, then the picture quality of theoutput image degrades.

Second Example of Configuration of Codec

FIG. 2 is a block diagram depicting a second example of a configurationof a codec that restores a texture lost by encoding.

Referring to FIG. 2, the codec includes an encoding apparatus 30 and adecoding apparatus 40.

The encoding apparatus 30 includes a texture DB (database) 31, a texturecomponent acquisition unit 32, a removal unit 33, an encoding unit 34and a transmission unit 35.

In the texture DB 31, texture components of various patterns, namely, aplurality of texture components (of a plurality of kinds of textures),are registered.

To the texture component acquisition unit 32, an original image as aninput image is supplied.

The texture component acquisition unit 32 acquires, for eachpredetermined block of an input image, a match component that is atexture component that best matches with the block from among thetexture components registered in the texture DB 31 and supplies theacquired match component to the removal unit 33. In particular, thetexture component acquisition unit 32 acquires a texture component that,for example, minimizes the sum total of square errors of pixel valuesfrom a predetermined block of the input image as a match component fromtexture components registered in the decoding unit 21 and supplies thematch component to the removal unit 33.

Furthermore, the texture component acquisition unit 32 suppliesidentification information for identifying the match component to thetransmission unit 35.

Here, in the texture DB 31, the texture component is registered togetherwith unique identification information for identifying the texturecomponent.

To the removal unit 33, not only the match component is supplied fromthe texture component acquisition unit 32, but also the input image issupplied.

The removal unit 33 arithmetically operates the difference between theinput image and the match component of the input image from the texturecomponent acquisition unit 32 to remove the texture component as thematch component of the input image, and supplies the input image, fromwhich the texture component is removed, as an encoding target image of atarget of encoding by the encoding unit 34 to the encoding unit 34.

Here, the encoding target image obtained from the removal unit 33 is animage obtained by removing the texture component, namely, a highfrequency component, from the input image, and therefore, it can beregarded that it is a low frequency component of the input image.

The encoding unit 34 encodes the encoding target image from the removalunit 33 by a non-reversible encoding method such as, for example, MPEG,AVC, HEVC or some other method and supplies encoded data obtained as aresult of the encoding to the transmission unit 35.

In FIG. 2, the encoding target image of the encoding unit 34 is theinput image from which the texture component is removed and efficientencoding can be performed, namely, the data mount of encoded data can bereduced, in comparison with that in the case where the input imageitself is an encoded target image.

Here, in the encoding of the non-reversible encoding method by theencoding unit 34, since an image is converted into a signal in thefrequency domain and the signal in the frequency domain is quantized, ahigh frequency component such as a texture component or the like is lostsimilarly as in the encoding unit 13 of FIG. 1.

The transmission unit 35 transmits identification information from thetexture component acquisition unit 32 and encoded data from the encodingunit 34. The identification information and the encoded data transmittedfrom the transmission unit 35 are supplied to the decoding apparatus 40through a transmission medium not depicted or is recorded on a recordingmedium not depicted and is then read out from the recording medium andsupplied to the decoding apparatus 40.

It is to be noted that the transmission unit 35 can transmit theidentification information and the encoded data separately from eachother and also it is possible for the transmission unit 35 to transmitthem integrally, namely, to multiplex and transmit the identificationinformation and the encoded data or the like.

Further, since the texture component acquisition unit 32 acquires, foreach predetermined block of the input image, a match component that is atexture component that best matches with the block, the match componentand hence the identification information can be obtained for each block.

The transmission unit 35 not only can transmit the identificationinformation for each block but also can transmit the identificationinformation in a unit of segmentation greater than a block, namely,collectively in a unit of a frame or the like.

The decoding apparatus 40 includes a reception unit 41, a decoding unit42, a texture DB 43, a texture component acquisition unit 44 and asynthesis unit 45.

The reception unit 41 accepts by receiving encoded data andidentification information transmitted from the transmission unit 35 orthe like, and supplies the encoded data to the decoding unit 42 andsupplies the identification information to the texture componentacquisition unit 44.

The decoding unit 42 decodes the encoded data from the reception unit 41by a method corresponding to the encoding method of the encoding unit 34and supplies a decoded image obtained as a result of the decoding to thesynthesis unit 45.

Here, the decoded image obtained by the decoding unit 42 corresponds toan encoding target image, namely, here, to a low frequency component ofthe input image.

In the texture DB 43, texture components of various patterns, namely, aplurality of texture components, are registered. For example, in thetexture DB 43, at least a plurality of texture components same as thoseregistered in the texture DB 31 of the encoding apparatus 30 areregistered.

The texture component acquisition unit 44 acquires a texture componentas a match component identified by the identification information fromthe reception unit 41 from among the texture components registered inthe texture DB 43 and supplies the acquired texture component to thesynthesis unit 45.

The synthesis unit 45 synthesizes the low frequency component of theinput image as a decoded image from the decoding unit 42 and the texturecomponent as a match component from the texture component acquisitionunit 44 with each other to generate a restoration image that restoresthe input image (original image) and outputs the generated restorationimage as an output image.

In the codec of FIG. 2, a plurality of texture components are registered(retained) in advance in the texture DBs 31 and 43 in the encodingapparatus 30 and the decoding apparatus 40, and identificationinformation for identifying a texture component as a match componentthat matches with the input image from among the plurality of texturecomponents is transmitted from the encoding apparatus 30 to the decodingapparatus 40.

Accordingly, the decoding apparatus 40 can obtain an output image thatrestores a texture of an input image using a texture component as amatch component identified by the identification information, namely,can improve the picture quality of the output image.

Furthermore, since, in the codec of FIG. 2, a plurality of texturecomponents are registered in advance in the texture DBs 31 and 43 in theencoding apparatus 30 and the decoding apparatus 40 and not a texturecomponent itself as a match component that matches with an input imagebut identification information for identifying the texture component istransmitted from the encoding apparatus 30 to the decoding apparatus 40,the transmission efficiency (compression efficiency) can be improved incomparison with that in the codec of FIG. 1 in which a texture componentitself is transmitted. Further, in the codec of FIG. 2, since an imageobtained by removing a texture component from an input image is made anencoding target image and the encoding target image is encoded by theencoding unit 34, the compression efficiency can be improved incomparison with that in the case where an input image itself is encodedby the encoding unit 34, and as a result, the transmission efficiencycan be improved.

As described above, with the codec of FIG. 2, the picture quality of anoutput image can be improved and the transmission efficiency can beimproved.

It is to be noted that, by increasing the number of texture componentsto be registered in advance into the texture DBs 31 and 43, the picturequality of an output image can be improved further without degrading thetransmission efficiency.

FIG. 3 is a flow chart illustrating an example of an encoding process ofthe encoding apparatus 30 of FIG. 2.

The encoding apparatus 30 successively determines frames of an inputimage supplied to the encoding apparatus 30 as a noticed frame andperforms an encoding process in accordance with the flow chart of FIG.3.

In particular, the texture component acquisition unit 32 divides(separates) a noticed frame of the input image into blocks for detectinga match component that matches with a texture component of the textureDB 31. Thus, at step S11, the texture component acquisition unit 32selects one block that has not yet been selected as a noticed block as anoticed block from among the blocks of the noticed frame of the inputimage, and the processing advances to step S12.

At step S12, the texture component acquisition unit 32 acquires a matchcomponent that is a texture component that best matches with the noticedblock of the input image from among the texture components registered inthe texture DB 31.

Here, the texture component acquisition unit 32 acquires, for example, atexture component that is most similar to the texture of the noticedblock among the texture components registered in the texture DB 31 as amatch component.

The texture component acquisition unit 32 supplies the match componentacquired in regard to the noticed block to the removal unit 33, and theprocessing advances to step S13.

At step S13, the texture component acquisition unit 32 acquiresidentification information for identifying the match component from thetexture DB 31 and supplies the identification information to thetransmission unit 35, and the processing advances to step S14.

At step S14, the texture component acquisition unit 32 decides whetheror not all blocks of the noticed frame of the input image have beenselected as a noticed block.

In the case where it is decided at step S14 that all blocks of thenoticed frame of the input image have not yet been selected as a noticedblock, the processing returns to step S11, and thereafter, similarprocesses are repeated.

On the other hand, in the case where it is decided at step S14 that allblocks of the noticed frame of the input image have been selected as anoticed block, the processing advances to step S15.

At step S15, the transmission unit 35 generates an identificationinformation map that associates the blocks of the noticed frame of theinput image and identification information of match components acquiredin regard to the blocks (identification information from the texturecomponent acquisition unit 32) with each other, and the processingadvances to step S16.

At step S16, the removal unit 33 generates a low frequency component of(the noticed frame of) the input image obtained by removing, from theblocks of the noticed frame of the input image, match components of theblocks from the texture component acquisition unit 32, namely, adifference between the input image and the texture components as thematch components, as an encoding target image and supplies the encodingtarget image to the encoding unit 34, and the processing advances tostep S17.

At step S17, the encoding unit 34 encodes the encoding target image fromthe removal unit 33 by a non-reversible encoding method and suppliesencoded data obtained as a result of the encoding to the transmissionunit 35, and the processing advances to step S18.

At step S18, the transmission unit 35 transmits the identificationinformation map and the encoded data from the encoding unit 34, and theencoding apparatus 30 ends the processing for the noticed frame of theinput image.

FIG. 4 is a flow chart illustrating an example of a decoding process ofthe decoding apparatus 40 of FIG. 2.

At step S21, the reception unit 41 receives (accepts) encoded data forone frame and an identification information map transmitted thereto fromthe encoding apparatus 30. Further, the reception unit 41 supplies theencoded data to the decoding unit 42 and supplies the identificationinformation map to the texture component acquisition unit 44, and theprocessing advances from step S21 to step S22.

At step S22, the decoding unit 42 decodes the encoded data from thereception unit 41 and supplies (a frame of) a decoded image obtained asa result of the decoding to the synthesis unit 45, and the processingadvances to step S23.

At step S23, the texture component acquisition unit 44 selects one ofthe blocks that have not yet been selected as a noticed block from amongthe blocks of the identification information map from the reception unit41 as a noticed block, and the processing advances to step S24.

At step S24, the texture component acquisition unit 44 acquires atexture component identified by the identification information(represented by the identification information) of the noticed block asa match component of the noticed block from the texture component of thetexture DB 43. Then, the texture component acquisition unit 44 suppliesthe match component of the noticed block to the synthesis unit 45, andthe processing advances from step S24 to step S25.

At step S25, the texture component acquisition unit 44 decides whetheror not all blocks of the identification information map have beenselected as a noticed block.

In the case it is decided at step S25 that all blocks of theidentification information map have not yet been selected as a noticedblock, the processing returns to step S23, and thereafter, similarprocesses are repeated.

On the other hand, in the case where it is decided at step S25 that allblocks of the identification information map have been selected as anoticed block, the processing advances to step S26.

At step S26, the synthesis unit 45 synthesizes the texture component asa match component for each block from the texture component acquisitionunit 44 at the corresponding position of the block of (the frame of) thedecoded image from the decoding unit 42 to generate (a frame of) arestoration image that restores the input image (original image) andoutputs the generated restoration image as an output image, therebyending the processing for the encoded data and the identificationinformation map for one frame.

Third Example of Configuration of Codec

FIG. 5 is a block diagram depicting a third example of a configurationof a codec that restores a texture lost by encoding.

It is to be noted that, in FIG. 5, elements corresponding to those ofthe case of FIG. 2 are denoted by same reference numerals, anddescription of them is hereinafter omitted suitably.

Referring to FIG. 5, the codec is common to that in the case of FIG. 2in that it includes an encoding apparatus 30 and a decoding apparatus40.

Furthermore, referring to FIG. 5, the encoding apparatus 30 is common tothat in the case of FIG. 2 in that it includes a texture DB 31, atexture component acquisition unit 32, an encoding unit 34 and atransmission unit 35, and the decoding apparatus 40 is common to that inthe case of FIG. 2 in that it includes a reception unit 41, a decodingunit 42, a texture DB 43, a texture component acquisition unit 44 and asynthesis unit 45.

However, in FIG. 5, the encoding apparatus 30 is different from that inthe case of FIG. 2 in that it does not include the removal unit 33.

Referring to FIG. 5, since the encoding apparatus 30 does not includethe removal unit 33, to the encoding unit 34, not the input image fromwhich a texture component is removed, namely, not a low frequencycomponent of the input image, but the input image itself is supplied asan encoding target image.

Accordingly, in FIG. 5, the encoding unit 34 encodes an input imageitself as an encoding target image.

The encoding unit 34 encodes the encoding target image by anon-reversible encoding method such as, for example, MPEG, AVC, HEVC orsome other method as described hereinabove with reference to FIG. 2.

With the encoding by the non-reversible encoding method of the encodingunit 34, a high frequency component of the input image as an encodingtarget image, namely, for example, at least part of a texture component,is lost.

However, in the decoding apparatus 40, the synthesis unit 45 synthesizesa decoded image from the decoding unit 42, namely, the input image fromwhich (at least part of) texture components are lost, and a texturecomponent as a match component from the texture component acquisitionunit 44 with each other.

Consequently, in the synthesis unit 45, a restoration image thatrestores the texture component lost by the encoding of the encoding unit34 is generated as an output image.

As described above, in the codec of FIG. 5, processing similar to thatof the codec of FIG. 2 is performed except that the encoding unit 34 ofthe encoding apparatus 30 performs encoding by a non-reversible encodingmethod using not a low frequency component of the input image but theinput image itself as an encoding target image.

Accordingly, with the codec of FIG. 5, the picture quality of the outputimage is improved and the transmission efficiency can be improvedsimilarly as in the case of FIG. 2.

Fourth Example of Configuration of Codec

FIG. 6 is a block diagram depicting a fourth example of a configurationof a codec that restores a texture lost by encoding.

Referring to FIG. 6, the codec includes an encoding apparatus 50 and adecoding apparatus 60.

The encoding apparatus 50 includes a texture DB 51, a separation unit52, a basis synthesis unit 53, a match component determination unit 54,a removal unit 55, an encoding unit 56 and a transmission unit 57.

In the texture DB 51, various kinds of texture components, namely, aplurality of texture components, are registered.

However, in the texture DB 51, each texture component is formed as abasis (basis learning) and is registered in the form of a basis. Thebasis of a texture component is an image component that can represent atexture by a linear combination of a finite number of bases.

To the separation unit 52, an original image as an input image issupplied. The separation unit 52 filters the input image to separate alow frequency component of the input image from the input image andsupplies the low frequency component to the basis synthesis unit 53.

The basis synthesis unit 53 performs, for each of a plurality of texturecontents whose basis is registered in the texture DB 51, basis synthesisof the low frequency component of the input image from the separationunit 52 and a basis registered in the texture DB 51.

The basis synthesis unit 53 generates, for each of a plurality oftexture components whose basis is registered in the texture DB 51, atexture component as a restoration component that restores a texturecomponent of the input image by basis synthesis and supplies thegenerated texture component to the match component determination unit54.

To the match component determination unit 54, not only a restorationcomponent regarding each of the plurality of texture components whosebasis is registered in the texture DB 51 is provided from the basissynthesis unit 53, but also the input image is inputted.

The match component determination unit 54 determines, for eachpredetermined block of the input image, a match component that is atexture component as a restoration component that best matches with theblock from among the restoration components from the basis synthesisunit 53 and supplies the determined match component to the removal unit55.

In particular, the match component determination unit 54 divides (aframe of) the input image into blocks for determination of a matchcomponent. As a block for determination of a match component, a block ofan arbitrary size such as, for example, a block with 16×16 pixels inlength and width or the like can be adopted.

Furthermore, the match component determination unit 54 determines, foreach block of the input image, a restoration component whose error withrespect to the block is in the minimum among the restoration componentsfrom the basis synthesis unit 53 as a match component.

As the error of a restoration component with respect to the block, forexample, the S/N, namely, the sum total of squares of differencesbetween the restoration components and the pixel values of the block(sum total of the square errors), the difference between predeterminedfeature amounts of an activity or the like between the restorationcomponent and the block or the like can be adopted.

Furthermore, the match component determination unit 54 suppliesidentification information for identifying the match components to thetransmission unit 57.

In particular, in the texture DB 51, bases of texture components areregistered together with unique identification information foridentifying the texture components similarly as in the texture DB 31 ofFIG. 2.

The match component determination unit 54 acquires identificationinformation (of the basis) of the texture component determined as amatch component in regard to each block of the input image from thetexture DB 51 and supplies the identification information to thetransmission unit 57.

As the identification information, a bit string whose size (data amount)is sufficiently smaller than that of a texture component such as, forexample, 6 bits or the like can be adopted. With 6 bits, bases of 64=26texture components can be identified in the maximum.

To the removal unit 55, not only a match component is supplied from thematch component determination unit 54, but also an input image issupplied.

The removal unit 55 removes a texture component as a match component ofthe input image from the input image by arithmetically operating thedifference between the input image and the match component (for eachblock) of the input image from the match component determination unit54, and supplies the input image, from which the texture component isremoved, as an encoding target image of a target of encoding by theencoding unit 56 to the encoding unit 56.

Here, since the encoding target image obtained by the removal unit 55 isan image obtained by removing a texture component, namely, a highfrequency component, from the input image, it can be regarded as a lowfrequency component of the input image.

The encoding unit 56 encodes the encoding target image from the removalunit 55 by a non-reversible encoding method such as, for example, MPEG,AVC, HEVC or some other method as a hybrid method that is a combinationof prediction coding and orthogonal transformation, and supplies encodeddata obtained as a result of the encoding to the transmission unit 57.

Here, in the encoding of the non-reversible encoding method by theencoding unit 56, since an image is converted into a signal of thefrequency domain and the signal of the frequency domain is quantizedsimilarly as in the encoding unit 13 of FIG. 1, a high frequencycomponent such as a texture component or the like is lost.

The transmission unit 57 transmits the identification information fromthe match component determination unit 54 and the encoded data from theencoding unit 56. The identification information and the encoded datatransmitted from the transmission unit 57 are supplied to the decodingapparatus 60 through a transmission medium not depicted or recorded on arecording medium not depicted and then read out from the recordingmedium and supplied to the decoding apparatus 60.

It is to be noted that the transmission unit 57 can transmit theidentification information and the encoded data not only separately fromeach other but also integrally with each other similarly to thetransmission unit 35 of FIG. 2.

Further, the transmission unit 57 not only can transmit theidentification information for each block but also can transmit theidentification information after it is collected in a unit of asegmentation greater than a block, namely, in a unit of a frame or thelike.

The decoding apparatus 60 includes a reception unit 61, a decoding unit62, a texture DB 63, a basis synthesis unit 64, a separation unit 65 anda synthesis unit 66.

The reception unit 61 receives (accepts) the encoded data and theidentification information transmitted from the transmission unit 57,and supplies the encoded data to the decoding unit 62 and supplies theidentification information to the basis synthesis unit 64.

The decoding unit 62 decodes the encoded data from the reception unit 61by a method corresponding to the encoding method of the encoding unit 56and supplies a decoded image obtained as a result of the decoding to theseparation unit 65 and the synthesis unit 66.

Here, the decoded image obtained by the decoding unit 62 corresponds tothe encoding target image, namely, a low frequency component of theinput image.

In the texture DB 63, various kinds of texture components, namely, aplurality of texture components, are registered. For example, in thetexture DB 63, at least a plural number of texture components equal tothe number of texture components registered in the texture DB 51 of theencoding apparatus 50 are registered.

It is to be noted that, in the texture DB 63, texture components areconverted into bases and are each registered in the form of a basissimilarly as in the texture DB 51.

To the basis synthesis unit 64, not only identification information issupplied from the reception unit 61 but also a low frequency componentof a decoded image is supplied from the separation unit 65.

The basis synthesis unit 64 acquires, for each block of the decodedimage corresponding to a block of the input image, a basis of a texturecomponent as a match component identified by the identificationinformation from the reception unit 61 from among the bases of thetexture components registered in the texture DB 63.

Furthermore, the basis synthesis unit 64 performs, for each block of thedecoded image, basis synthesis using the low frequency component of thedecoded image supplied from the separation unit 65 and the basesacquired from the texture DB 63 similarly to the basis synthesis unit53. The basis synthesis unit 64 generates, for each block of the decodedimage, a texture component as a restored component restored from thematch component as a texture component of the input image by basissynthesis and supplies the generated texture component to the synthesisunit 66.

The separation unit 65 filters the decoded image from the decoding unit62 to separate a low frequency component of the decoded image from thedecoded image and supplies the low frequency component to the basissynthesis unit 64.

Here, the pass bands of the filtering performed by the separation units52 and 65 are, for example, same as each other.

Then, since, in FIG. 6, the decoded image is (an image corresponding to)a low frequency component of the input image, by the separation unit 65,an image substantially similar to the decoded image is obtained as a lowfrequency component of the decoded image by a degree that distortionoccurring with the decoded image is removed.

Accordingly, in FIG. 6, the separation unit 65 is not essentiallyrequired, and the decoding apparatus 60 can be configured without theseparation unit 65.

The synthesis unit 66 generates a restoration image that restores theinput image (original image) by synthesizing a low frequency componentof the input image as the decoded image from the decoding unit 62 andthe texture component as a match component from the basis synthesis unit64 and outputs the restoration image as an output image.

In the codec of FIG. 6, a plurality of texture components are registered(retained) in advance in the texture DBs 51 and 63 in the encodingapparatus 50 and the decoding apparatus 60, and identificationinformation for identifying the texture component as a match componentthat matches with the input image from among the plurality of texturecomponents is transmitted from the encoding apparatus 50 to the decodingapparatus 60.

Accordingly, the decoding apparatus 60 can obtain an output image thatrestores the texture of the image signal, namely, can improve thepicture quality of the output image, using a texture component as amatch component identified by the identification information.

Furthermore, in the codec of FIG. 6, a plurality of texture componentsare registered in advance in the texture DBs 51 and 63 in the encodingapparatus 50 and the decoding apparatus 60, and not a texture componentitself as a match component that matches with the input image butidentification information for identifying the texture component istransmitted from the encoding apparatus 50 to the decoding apparatus 60.Therefore, the transmission efficiency (compression efficiency) can beimproved in comparison with the codec of FIG. 1 in which the texturecomponent itself is transmitted. Further, in the codec of FIG. 6, sincean image obtained by removing a texture component from an input image isdetermined as an encoding target image and the encoding target image isencoded by the encoding unit 56, the compression efficiency can beimproved in comparison with that in the case where the input imageitself is encoded by the encoding unit 56, and as a result, thetransmission efficiency can be improved.

In this manner, with the codec of FIG. 6, the picture quality of anoutput image can be improved and the transmission efficiency can beimproved similarly to the codec of FIG. 2.

Furthermore, in the codec of FIG. 6, texture components are registeredeach in the form of a basis in the texture DBs 51 and 63. Accordingly,the capacities necessary for the texture DBs 51 and 63 can be reduced incomparison with those in an alternative case in which texture componentsare registered as they are in the form of an image.

Further, by registering the texture components each in the form of abasis, texture components of various patterns can be generated in regardto each kind of texture.

In particular, textures can be classified, for example, depending uponbodies (objects) that have the texture such as, for example, a forest, arock, water (water surface), a cloth and so forth.

Further, as a texture of each kind such as a forest or the like,textures of various patterns are available.

In the case where texture components of various patterns are preparedfor a texture of each kind and they are registered as they are (in theform of an image) into the texture DBs 51 and 63, a great capacity isrequired for the texture DBs 51 and 63.

In contrast, by registering texture components of each kind in the formof bases into the texture DBs 51 and 63, the capacity necessary for thetexture DBs 51 and 63 can be reduced significantly.

Further, by performing basis synthesis using a basis of a texturecomponent, texture components of various patterns can be generated for atexture of each kind.

<Basis Learning and Basis Synthesis>

FIG. 7 is a view illustrating an overview of basis learning forconverting a texture component into a basis.

In the basis learning, for each kind of a texture such as a forest, arock, water, a cloth or the like, an image as a texture component isprepared as a learning image for learning, and the learning image isused to determine a basis of the texture component for each kind.

In particular, by setting the learning image to a high resolution imageof a high resolution and performing filtering of the high resolutionimage (for example, filtering similar to that by the separation unit 52or 65 of FIG. 6), a low resolution image of a low resolution, which is alow frequency component of the high resolution image, is determined.

Furthermore, basis learning is performed for a pair of images includingthe high resolution image as the learning image and the low resolutionimage determined from the high resolution image to determine a pair ofbases including a high resolution basis of a high resolution and a lowresolution basis of a low resolution. Each basis as a high resolutionbasis and each basis as a low resolution basis correspond to each other.In other words, among high resolution bases, a basis corresponding to(paired with) each basis as a low resolution basis exists.

As a method for basis learning, for example, the k-SVD method, theK-means method and so forth can be adopted.

FIG. 8 is a view illustrating an overview of basis synthesis forgenerating a texture component using a basis.

The basis synthesis can be performed, for example, by the method ofMatching Pursuits.

Now, it is assumed that a certain image is made a noticed image and alow frequency component of the noticed image formed by removing part orall of a texture component from the noticed image is made a restorationtarget image of a target for restoration of a texture.

In the basis synthesis, a block whose texture is to be restored issuccessively selected as a noticed block, for example, in a raster scanorder from within a restoration target image.

Furthermore, from among bases as low resolution bases of a certain kindof texture, a plurality of bases are selected as selection bases, and aprediction block that predicts (an image of) the noticed block isgenerated by linear coupling of the selection bases.

In the basis synthesis, bases of a combination that can be selected fromamong low resolution bases are successively selected as selection bases,and coefficients w0, w1, . . . to be used for linear coupling of theselection bases are determined such that a prediction block determinedby the linear coupling of the selection bases becomes coincident withthe noticed block as far as possible, namely, for example, such that theerror between the prediction block and the noticed block becomesminimized statistically.

It is to be noted that the number of coefficients wi coincides with thenumber of selection bases. Further, in the case where the coefficient wiis to be determined for all combinations of selection bases that can beselected from among the low resolution bases, since the arithmeticoperation cost required for the determinations of the coefficients wimay become very great, the maximum number of bases to be selected as theselection basis can be restricted to a predetermined value such that thecoefficient wi is determined only for combinations of selection baseswithin the range of the restriction.

After the coefficients wi are determined for the combinations ofselection bases selected from the low resolution bases, the coefficientwi whose error of the prediction block from the noticed block issmallest is determined as a generation coefficient wi for texturecomponent generation.

Furthermore, the high resolution basis paired with the low resolutionbasis as the selection basis when the prediction block whose error issmallest is obtained is determined as a generation basis for texturecomponent generation.

Then, a texture component of the noticed block of the texturerestoration image that restores the texture (accurately, a block of thetexture restoration image at a position same as the position of thenoticed block of the restoration target image) is determined by linearcoupling of the generation basis and the generation coefficient wi,namely, by product sum operation between the generation basis and thegeneration coefficient wi.

It is to be noted that the basis synthesis can be performed not only bythe method of Matching Pursuits but also, for example, by IterativeReweighted Least Square or the like. Further, basis learning and basissynthesis are described, for example, in Jianchao Yang J., Huang T. S.,Yi MaWright. (2010). Image Super-Resolution via Sparse Representation.Image processing, IEEE Transaction, Vol. 19, Issue 11, pp. 2861-2873.

FIG. 9 is a flow chart illustrating an example of an encoding process ofthe encoding apparatus 50 of FIG. 6.

The encoding apparatus 50 successively determines frames of an inputimage supplied to the encoding apparatus 50 as a noticed frame andperforms an encoding process in accordance with the flow chart of FIG. 9for the noticed frame.

In particular, at step S41, the separation unit 52 separates a lowfrequency component from the noticed frame of the input image andsupplies the low frequency component to the basis synthesis unit 53.

The basis synthesis unit 53 divides the noticed frame of the input imageinto blocks for determination of a match component. Then, the processingadvances from step S41 to step S42, at which the basis synthesis unit 53selects one block that has not been selected as a noticed block as yetfrom among the blocks of the noticed frame of the input image as anoticed block. Then, the processing advances to step S43.

At step S43, the basis synthesis unit 53 selects one texture componentthat has not been selected as a noticed component from among theplurality of (kinds of) texture components whose basis is registered(stored) in the texture DB 51 as a noticed component, and the processingadvances to step S44.

At step S44, the basis synthesis unit 53 performs basis synthesis usingthe low frequency component of the noticed block among the low frequencycomponents of the input image from the separation unit 52 and the basisof the noticed component to determine a restoration component thatrestores the texture component of the noticed block.

The basis synthesis unit 53 supplies the restoration component of thenoticed block to the match component determination unit 54, and theprocessing advances from step S44 to step S45.

At step S45, the match component determination unit 54 calculates anerror of the restoration component of the noticed block with respect tothe noticed block of the input image, and the processing advances tostep S46.

At step S46, the match component determination unit 54 decides whetheror not the error of the restoration component of the noticed block issmaller than a minimum error regarding the noticed block.

Here, the minimum error regarding the noticed block is a minimum valueamong errors of the restoration components of the noticed blocksdetermined in regard to the texture components that have been selectedas a texture component up to now from among the texture components whosebasis is registered in the texture DB 51, and as an initial value of theminimum error regarding the noticed block, a predetermined high value isadopted.

In the case where it is decided at step S46 that the error of therestoration component of the noticed block is not smaller than theminimum error regarding the noticed block, the processing skips step S47and advances to step S48.

On the other hand, in the case where it is decided at step S46 that theerror of the restoration component of the noticed block is smaller thanthe minimum error regarding the noticed block, the processing advancesto step S47.

At step S47, the match component determination unit 54 updates theminimum error regarding the noticed block to the error of therestoration component of the noticed block, namely, to the latest error,and the processing advances to step S48.

At step S48, the match component determination unit 54 decides whetheror not the error of the restoration component of the noticed block hasbeen acquired in regard to all texture components whose basis isregistered in the texture DB 51.

In the case where it is decided at step S48 that the error of therestoration component of the noticed block has not yet been acquired inregard to all texture components whose basis is registered in thetexture DB 51, the processing returns to step S43.

In particular, in the case where a texture component that has not yetbeen selected as a noticed component exists in the texture componentswhose basis is registered in the texture DB 51, the processing returnsfrom step S48 to step S43, and thereafter, the processes described aboveare repeated.

On the other hand, in the case where it is decided at step S48 that theerror of the restoration component of the noticed block has beenacquired in regard to all texture components whose basis is registeredin the texture DB 51, the processing advances to step S49.

In particular, in the case where an error of a restoration component ofa noticed block has been determined in regard to all texture componentswhose basis is registered in the texture DB 51 and a minimum error amongthe errors is determined as a minimum value regarding the noticed block,the processing advances from step S48 to step S49.

At step S49, the match component determination unit 54 determines thetexture component from which the minimum error is obtained in regard tothe noticed block among the texture components whose basis is registeredin the texture DB 51 as a match component. Further, the match componentdetermination unit 54 acquires and supplies identification informationof the match component to the transmission unit 57, and the processingadvances from step S49 to step S50.

At step S50, the basis synthesis unit 53 decides whether or not allblocks of the noticed frame of the input image have been selected as anoticed block.

In the case where it is decided at step S50 that all blocks of thenoticed frame of the input image have not yet been selected as a noticedblock, the processing returns to step S42, and thereafter, similarprocesses are repeated.

On the other hand, in the case where it is decided at step S50 that allblocks of the noticed frame of the input image have been selected as anoticed block, the processing advances to step S51.

At step S51, the transmission unit 57 generates an identificationinformation map that associates the blocks of the noticed frame of theinput image and the identification information of the match componentsof the blocks (identification information from the match componentdetermination unit 54) with each other, and the processing advances tostep S52.

At step S52, the removal unit 55 generates a low frequency component (ofthe noticed frame) of the input image obtained by removing a matchcomponent of each block from the basis synthesis unit 53 from each blockof the noticed frame of the input image, namely, a difference betweenthe input image and a texture component as a match component as anencoding target image and supplies the encoding target image to theencoding unit 56, and the processing advances to step S53.

At step S53, the encoding unit 56 encodes the encoding target image fromthe removal unit 55 by the non-reversible encoding method and suppliesencoded data obtained as a result of the encoding to the transmissionunit 57, and the processing advances to step S54.

At step S54, the transmission unit 57 transmits the identificationinformation map and the encoded data from the encoding unit 56, and theencoding apparatus 50 ends the processing for the notice frame of theinput image.

FIG. 10 is a flow chart depicting an example of a decoding process ofthe decoding apparatus 60 of FIG. 6.

At step S61, the reception unit 61 receives (accepts) encoded data forone frame and the identification information map transmitted from theencoding apparatus 50. Further, the reception unit 61 supplies theencoded data to the decoding unit 62 and supplies the identificationinformation map to the basis synthesis unit 64, and the processingadvances from step S61 to step S62.

At step S62, the decoding unit 62 decodes the encoded data from thereception unit 61 and supplies (a frame of) a decoded image obtained asa result of the decoding to the separation unit 65 and the synthesisunit 66, and the processing advances to step S63.

At step S63, the separation unit 65 separates a low frequency componentof the decoded image from the decoded image from the decoding unit 62and supplies the separated low frequency component to the basissynthesis unit 64, and the processing advances to step S64.

At step S64, the basis synthesis unit 64 selects one block that has notbeen selected as a noticed block as yet among the blocks of theidentification information map from the reception unit 61 as a noticedblock, and the processing advances to step S65.

At step S65, the basis synthesis unit 64 acquires the basis of a texturecomponent identified by the identification information (represented bythe identification information) of the noticed block from among thebases of the texture components of the texture DB 63 as a basis of anoticed component, and the processing advances to step S66.

At step S66, the basis synthesis unit 64 performs basis synthesissimilar to that by the basis synthesis unit 53 using the low frequencycomponent of the noticed block among the low frequency components of thedecoded image from the separation unit 65 and the basis of the noticedcomponent to restore a texture component as a match component of thenoticed block.

Then, the basis synthesis unit 64 supplies the match component of thenoticed block to the synthesis unit 66, and the processing advances fromstep S66 to step S67.

At step S67, the basis synthesis unit 64 decides whether or not allblocks of the identification information map have been selected as anoticed block.

In the case where it is decided at step S67 that all blocks of theidentification information map have not yet been selected as a noticedblock, the processing returns to step S64, and thereafter, similarprocesses are repeated.

On the other hand, in the case where it is decided at step S67 that allblocks of the identification information map have been selected as anoticed block, the processing advances to step S68.

At step S68, the synthesis unit 66 synthesizes the texture component asa match component for each block from the basis synthesis unit 64 at acorresponding position of the block of (the frame of) the decoded imagefrom the decoding unit 62 to generate (a frame of) a restoration imagethat restores the input image (original image), and outputs therestoration image as an output image, thereby ending the processing forthe encoded data for one frame and the identification information map.

Fifth Example of Configuration of Codec

FIG. 11 is a block diagram depicting a fifth example of a configurationof a codec that restores a texture lost by encoding.

It is to be noted that, in FIG. 11, elements corresponding to those ofthe case of FIG. 6 are denoted by same reference numerals, anddescription of them is hereinafter omitted suitably.

In FIG. 11, the codec is common to that of the case of FIG. 6 in that itincludes an encoding apparatus 50 and a decoding apparatus 60.

Furthermore, referring to FIG. 11, the encoding apparatus 50 is commonto that in the case of FIG. 6 in that it includes the elements from thetexture DB 51 to the match component determination unit 54, the encodingunit 56 and the transmission unit 57, and the decoding apparatus 60 iscommon to that in the case of FIG. 6 in that it includes the elementsfrom the reception unit 61 to the synthesis unit 66.

However, in FIG. 11, the encoding apparatus 50 is different from that inthe case of FIG. 6 in that it does not include the removal unit 55.

Referring to FIG. 11, since the encoding apparatus 50 does not includethe removal unit 55, to the encoding unit 56, not an input image fromwhich a texture component is removed, namely, not a low frequencycomponent of the input image, but the input image itself is supplied asan encoding target image.

Accordingly, in FIG. 11, the encoding unit 56 encodes an input imageitself as an encoding target image.

By the encoding unit 56, an encoding target image is encoded by anon-reversible encoding method such as, for example, MPEG, AVC, HEVC orsome other method as described hereinabove with reference to FIG. 6.

With the encoding by the non-reversible encoding method of the encodingunit 56, at least part of a texture component of an input image itselfas the encoding target image is lost.

However, in the decoding apparatus 60, the synthesis unit 66 synthesizesa decoded image from the decoding unit 62, namely, an input image fromwhich (at least part of) a texture component is lost, and a texturecomponent as a match component from the basis synthesis unit 64.

Consequently, in the synthesis unit 66, a restoration image thatrestores the texture component lost by the encoding of the encoding unit56 is generated as an output image.

As described above, in the codec of FIG. 11, processing similar to thatby the codec of FIG. 6 is performed except that, in the encoding unit 56of the encoding apparatus 50, encoding by a non-reversible encodingmethod is performed using not a low frequency component of an inputimage but the input image itself as an encoding target image.

Accordingly, with the codec of FIG. 11, the picture quality of theoutput image is improved and the transmission efficiency can be improvedsimilarly as in the case of FIG. 6.

Here, in the codec of FIG. 11, since a decoded image and a texturecomponent as a match component are synthesized by the decoding apparatus60, a restoration image that restores the texture lost in the decodingof the encoding unit 56 is generated as an output image. In other words,even if such high compression encoding as may remove many texturecomponents is performed by the encoding unit 56, the decoding apparatus60 can restore the removed texture components.

Sixth Example of Configuration of Codec

FIG. 12 is a block diagram depicting a sixth example of a configurationof a codec that restores a texture lost by encoding.

It is to be noted that, in FIG. 12, elements corresponding to those ofthe case of FIG. 6 are denoted by same reference numerals, anddescription of them is hereinafter omitted suitably.

In FIG. 12, the codec is common to that of the case of FIG. 6 in that itincludes an encoding apparatus 50 and a decoding apparatus 60.

Further, referring to FIG. 12, the encoding apparatus 50 is common tothat in the case of FIG. 6 in that it includes the texture DB 51, basissynthesis unit 53, match component determination unit 54, encoding unit56 and transmission unit 57, and the decoding apparatus 60 is common tothat in the case of FIG. 6 in that it includes the elements from thereception unit 61 to the basis synthesis unit 64 and the synthesis unit66.

However, in FIG. 12, the encoding apparatus 50 is different from that inthe case of FIG. 6 in that it does not include the separation unit 52nor the removal unit 55. Further, in FIG. 12, the encoding apparatus 50is different from that in the case of FIG. 6 in that it newly includes adecoding unit 81.

Further, in FIG. 12, the decoding apparatus 60 is different from that inthe case of FIG. 6 in that it does not include the separation unit 65.

As described above, in FIG. 12, since the encoding apparatus 50 does notinclude the removal unit 55, in the encoding apparatus 50, not the inputimage from which a texture component is removed in the encoding unit 56,namely, not a low frequency component of the input image, but the inputimage itself is encoded as an encoding target image similarly as in thecase of FIG. 11.

Further, in the encoding apparatus 50 of FIG. 12, to the decoding unit81, encoded data obtained by encoding an input image itself as anencoding target image by the encoding unit 56 is supplied.

The decoding unit 81 decodes the encoded data from the encoding unit 56similarly to the decoding unit 62 and supplies a decoded image obtainedas a result of the decoding to the basis synthesis unit 53.

Here, as described with reference to FIG. 11, with the encoding by thenon-reversible encoding method of the encoding unit 56, at least part ofa texture component of the input image itself as the encoding targetimage is lost.

Accordingly, the decoded image obtained by the decoding unit 81 is animage from which the texture component of the input image is lost,namely, an image corresponding to the low frequency component of theinput image.

In FIG. 12, the basis synthesis unit 53 performs basis synthesis using adecoded image corresponding to the low frequency component of the inputimage in place of the low frequency component itself of the input image.

As described above, in FIG. 12, in accordance with the basis synthesisperformed by the basis synthesis unit 53 in the encoding apparatus 50using a decoded image corresponding to a low frequency component of aninput image, also in the decoding apparatus 60, the basis synthesis unit64 performs basis synthesis using a decoded image obtained by thedecoding unit 62.

Therefore, in FIG. 12, the decoding apparatus 60 is configured such thatit does not include the separation unit 65 of FIG. 6 (and FIG. 11).

FIG. 13 is a flow chart illustrating an example of an encoding processof the encoding apparatus 50 of FIG. 12.

The encoding apparatus 50 successively determines frames of an inputimage supplied to the encoding apparatus 50 as a noticed frame andperforms an encoding process in accordance with the flow chart of FIG.13 for the noticed frame.

In particular, at step S71, the encoding unit 56 encodes a noticed frameof an input image as an encoded target image by a non-reversibleencoding method and supplies encoded data obtained as a result of theencoding to the transmission unit 57 and the decoding unit 81, and theprocessing advances to step S72.

At step S72, the decoding unit 81 decodes the encoded data of thenoticed frame and supplies a decoded image obtained as a result of thedecoding and corresponding to a low frequency component of the noticedframe of the input image to the basis synthesis unit 53.

The basis synthesis unit 53 divides the decoded image of the noticedframe into blocks for determination of match components. Then, theprocessing advances from step S72 to step S73, at which the basissynthesis unit 53 selects one block that has not been selected as anoticed block as yet from among the blocks of the decoded image of thenoticed frame as a noticed block, and the processing advances to stepS74.

At step S74, the basis synthesis unit 53 selects one texture componentthat has not been selected as a noticed component as yet from among aplurality of (kinds of) texture components whose basis is registered inthe texture DB 51 as a noticed component, and the processing advances tostep S75.

At step S75, the basis synthesis unit 53 performs basis synthesis usingthe noticed block in the encoded image from the decoding unit 81 and thebasis of the noticed component to determine a restoration component thatrestores the texture component of the input image (accurately, the blockof the input image at a position same as that of the noticed block ofthe decoded image).

The basis synthesis unit 53 supplies the restoration component of thenoticed block of the input image to the match component determinationunit 54, and the processing advances from step S75 to step S76.

At step S76, the match component determination unit 54 calculates anerror of the restoration component of the noticed block of the inputimage with respect to the noticed block of the input image, and theprocessing advances to step S77.

At step S77, the match component determination unit 54 decides whetheror not the error of the restoration component of the noticed block ofthe input image is smaller than a minimum error regarding the noticedblock.

Here, the minimum error regarding the noticed block is a minimum valueamong errors of the restoration component of the noticed block of theinput image determined with regard to texture components selected as anoticed component up to now from among the texture components whosebasis is registered in the texture DB 51 similarly as in the casedescribed hereinabove with reference to FIG. 9, and a predetermined highvalue is adopted as an initial value of the minimum error for a noticedblock is adopted.

In the case where it is decided at step S77 that the error of therestoration component of the noticed block of the input image is notsmaller than a minimum error regarding the noticed block, the processingskips step S78 and advances to step S79.

On the other hand, in the case where it is decided at step S77 that theerror of the restoration component of the noticed block of the inputimage is smaller than a minimum error regarding the noticed block, theprocessing advances to step S78.

At step S78, the match component determination unit 54 updates theminimum error regarding the noticed block to the error of therestoration component of the noticed block of the input image, namely,to the latest error, and the processing advances to step S79.

At step S79, the match component determination unit 54 decides whetheror not the error of the restoration component of the noticed block ofthe input image has been acquired in regard to all texture componentswhose basis is registered in the texture DB 51.

In the case where it is decided at step S79 that the error of therestoration component of the noticed block of the input image has notyet been acquired in regard to all texture components whose basis isregistered in the texture DB 51, the processing returns to step S74.

In particular, in the case where a texture component that has not beenselected as a noticed component exists among the texture componentswhose basis is registered in the texture DB 51, the processing returnsfrom step S79 to step S74, and thereafter, the processes describedhereinabove are repeated.

On the other hand, in the case where it is decided at step S79 that theerror of the restoration component of the noticed block of the inputimage has been acquired in regard to all texture components whose basisis registered in the texture DB 51, the processing advances to step S80.

In particular, in the case where an error of a restoration component ofa noticed block of the input image has been determined in regard to alltexture components whose basis is registered in the texture DB 51 and aminimum error among the errors is determined as a minimum errorregarding the noticed block, the processing advances from step S79 tostep S80.

At step S80, the match component determination unit 54 determines, fromamong the texture components as restoration components of the noticedblock of the input image, a texture component from which a minimum valueregarding a noticed block is obtained as a match component. Furthermore,the match component determination unit 54 acquires and suppliesidentification information of the match component to the transmissionunit 57, and the processing advances from step S80 to step S81.

At step S81, the basis synthesis unit 53 decides whether or not allblocks of the decoded image of the noticed frame have been selected as anoticed block.

In the case where it is decided at step S81 that all blocks of thedecoded image of the noticed frame have not yet been selected as anoticed block, the processing returns to step S73, and thereafter,similar processes are repeated.

On the other hand, in the case where it is decided at step S81 that allblocks of the decoded image of the noticed frame have been selected as anoticed block, the processing advances to step S82.

At step S82, the transmission unit 57 generates an identificationinformation map that associates the blocks of the noticed frame of theinput image and the identification information of the match componentsof the blocks (identification information from the transmission unit 57)with each other, and the processing advances to step S83.

At step S83, the transmission unit 57 transmits the identificationinformation map and the encoded data from the encoding unit 56, and theencoding apparatus 50 ends the processing for the noticed image of theinput image.

FIG. 14 is a flow chart illustrating an example of a decoding process ofthe decoding apparatus 60 of FIG. 12.

At step S91, the reception unit 61 receives (accepts) encoded data forone frame and an identification information map transmitted from theencoding apparatus 50. Further, the reception unit 61 supplies theencoded data to the decoding unit 62 and supplies the identificationinformation map to the basis synthesis unit 64, and the processingadvances from step S91 to step S92.

At step S92, the decoding unit 62 decodes the encoded data from thereception unit 61 and supplies (a frame of) a decoded image obtained asa result of the decoding to the basis synthesis unit 64 and thesynthesis unit 66, and the processing advances to step S93.

At step S93, the basis synthesis unit 64 selects one block that has notbeen selected as a noticed block as yet among the blocks of theidentification information map from the reception unit 61 as a noticedblock, and the processing advances to step S94.

At step S94, the basis synthesis unit 64 acquires a basis of a texturecomponent identified by the identification information of the noticedblock (represented by the identification information) from among thebases of the texture components of the texture DB 63 as a basis of thenoticed component, and the processing advances to step S95.

At step S95, the basis synthesis unit 64 performs basis synthesissimilar to that by the basis synthesis unit 53 using the noticed blockin the decoded image from the separation unit 65 (accurately a block ofthe decoded image at a position similar to that of the noticed blockfrom among the blocks of the identification information map) to restorea texture component as a match component of the noticed block.

Then, the basis synthesis unit 64 supplies the match component of thenoticed block to the synthesis unit 66, and the processing advances fromstep S95 to step S96.

At step S96, the basis synthesis unit 64 decides whether or not allblocks of the identification information map have been selected as anoticed block.

In the case where it is decided at step S96 that all blocks of theidentification information map have not been selected as a noticed blockas yet, the processing returns to step S93, and thereafter, similarprocesses are repeated.

On the other hand, in the case where it is decided at step S96 that allblocks of the identification information map have been selected as anoticed block, the processing advances to step S97.

At step S97, the synthesis unit 66 synthesizes the texture component asa match component for each block from the basis synthesis unit 64 at acorresponding position of the block (of the frame) of the decoded imagefrom the decoding unit 62 to generate (a frame of) a restoration imagethat restores the input image (original image) and outputs the generatedrestoration image as an output image, thereby ending the processing forthe encoded data for one frame and the identification information map.

Seventh Example of Configuration of Codec

FIG. 15 is a block diagram depicting a seventh example of aconfiguration of a codec that restores a texture lost by encoding.

Here, in the codec, (a basis of) a texture component of the texture DB51 or 63 can be updated as occasion demands.

In the following, an example of a configuration of the codec in whichthe texture components in the texture DBs 51 and 63 can be updated isdescribed taking an example in which a function for updating a texturecomponent is added to the codec of the fourth example of a configurationof FIG. 6 as an example.

It is to be noted that the function for updating a texture component canbe added to the codecs of the examples of a configuration other than thefourth example of a configuration of FIG. 6 (for example, to the secondconfiguration example of FIG. 2, third configuration example of FIG. 6,fifth configuration example of FIG. 11, sixth configuration example ofFIG. 12 and so forth)

FIG. 15 depicts an example of a configuration of a codec in whichupdating of a basis of a texture component of the texture DB 63 can beperformed.

It is to be noted that, in FIG. 15, elements corresponding to those ofthe case of FIG. 6 are denoted by same reference numerals, anddescription of them is hereinafter omitted suitably.

Referring to FIG. 15, the codec is common to that in the case of FIG. 6in that it includes an encoding apparatus 50 and a decoding apparatus60.

Further, in FIG. 15, the encoding apparatus 50 is common to that in thecase of FIG. 6 in that it includes the elements from the texture DB 51to the transmission unit 57, and the decoding apparatus 60 is common tothat in the case of FIG. 6 in that it includes the elements from thereception unit 61 to the synthesis unit 66.

However, in FIG. 15, the encoding apparatus 50 is different from that inthe case of FIG. 6 in that it newly includes a data transmission unit101. Further, in FIG. 15, the decoding apparatus 60 is different fromthat in the case of FIG. 6 in that it newly includes an updating unit111.

The data transmission unit 101 transmits a basis (and identificationinformation) of a texture component as DB data registered in the textureDB 51 in response to a request from the updating unit 111 of thedecoding apparatus 60.

The updating unit 111 refers to the bases of texture components as DBdata registered in the texture DB 51 through the data transmission unit101 of the encoding apparatus 50 to confirm whether or not a basis of atexture component that is not yet registered in the texture DB 63 of thedecoding apparatus 60 (hereinafter referred to as unregisteredcomponent) exists in the bases of texture components registered in thetexture DB 51.

Then, in the case where a basis of an unregistered component exists, theupdating unit 111 requests the data transmission unit 101 of theencoding apparatus 50 and acquires the basis of the unregisteredcomponent, and registers the basis of the unregistered component intothe texture DB 63 to update the DB data as the registration substance ofthe texture DB 63.

It is to be noted that the basis of the unregistered component includesnot only a basis of a texture component of a different kind but also anew basis of a texture component that is obtained by new basis learning(a so-to-speak basis of a new version) even if the kind of the texturecomponent is same.

FIG. 16 is a flow chart illustrating an example of processing performedby the encoding apparatus 50 and the decoding apparatus 60 in order toupdate the DB data of the texture DB 63.

In the encoding apparatus 50, at step S111, the data transmission unit101 decides whether or not a request for DB data is received from theupdating unit 111 of the decoding apparatus 60.

In the case where it is decided at step S111 that a request for DB datais no received, the processing returns to step S111.

On the other hand, in the case where it is decided at step S111 that arequest for DB data is received, the processing advances to step S112.At step S112, the data transmission unit 101 acquires the basis (and theidentification) of the texture component as DB data for which therequest is received from the updating unit 111 from the texture DB 51and transmits the acquired basis (and identification) to the updatingunit 111. Then, the processing returns from step S112 to step S111, andthereafter, similar processes are repeated.

On the other hand, in the decoding apparatus 60, it is decided at stepS121 whether or not the updating unit 111 is at an update timing of thetexture DB 63.

Here, as the update timing of the texture DB 63, for example, a timingimmediately before the decoding unit 62 starts decoding of a certaincontent, a timing immediately before the decoding unit 62 startsprocessing of a block after decoding is started, a timing immediatelybefore the decoding unit 62 starts processing of a frame, a periodicalor non-periodical timing or some other arbitrary timing can be adopted.

In the case where it is decided at step S121 that the updating unit 111is not at an update timing of the texture DB 63, the processing returnsto step S121.

On the other hand, in the case where it is decided at step S121 that theupdating unit 111 is at an update timing of the texture DB 63, theprocessing advances to step S122.

At step S122, the updating unit 111 decides whether or not unregisteredDB data, namely, a basis of an unregistered component that is notregistered in the texture DB 63, exists in the bases of texturecomponents registered in the texture DB 51.

In the case where it is decided at step S122 that a basis of anunregistered component does not exist, the processing returns to stepS121.

On the other hand, in the case where it is decided at step S122 that abasis of an unregistered component exists, the processing advances tostep S123.

At step S123, the updating unit 111 requests the data transmission unit101 of the encoding apparatus 50 for the unregistered DB data, namely,for the basis of the unregistered component, and the processing advancesto step S124.

At step S124, the updating unit 111 waits that the basis of theunregistered component is transmitted from the data transmission unit101 and receives the basis of the unregistered component. Furthermore,the updating unit 111 registers the basis of the unregistered componentfrom the data transmission unit 101 into the texture DB 63 to update theDB data of the texture DB 63, and the processing returns from step S124to step S121.

In this manner, in the case where a basis of an unregistered componentthat is registered in the texture DB 51 of the encoding apparatus 50 butis not registered in the texture DB 63 of the decoding apparatus 60exists, the decoding apparatus 60 acquires the basis of the unregisteredcomponent from the encoding apparatus 50 and registers the basis of theunregistered component into the texture DB 63 such that the DB data ofthe texture DB 51 of the encoding apparatus 50 are included in the DBdata of the texture DB 63 of the decoding apparatus 60.

Eighth Example of Configuration of Codec

FIG. 17 is a block diagram depicting an eighth example of aconfiguration of a codec that restores a texture lost by encoding.

In particular, an example of a configuration of the codec in which thebases of the texture components in the texture DBs 51 and 63 can beupdated.

It is to be noted that, in FIG. 17, elements corresponding to those ofthe case of FIG. 6 are denoted by same reference numerals, anddescription of them is hereinafter omitted suitably.

In FIG. 17, the codec is common to that of the case of FIG. 6 in that itincludes an encoding apparatus 50 and a decoding apparatus 60.

Further, referring to FIG. 17, the encoding apparatus 50 is common tothat in the case of FIG. 6 in that it includes the elements from thetexture DB 51 to the transmission unit 57, and the decoding apparatus 60is common to that in the case of FIG. 6 in that it includes the elementsfrom the reception unit 61 to the synthesis unit 66.

However, in FIG. 17, the encoding apparatus 50 is different from that inthe case of FIG. 6 in that it newly includes an updating unit 121.Further, in FIG. 17, the decoding apparatus 60 is different from that inthe case of FIG. 6 in that it newly includes an updating unit 131.

The updating unit 121 accesses an external server 141 such as a serveron the Internet or the like as occasion demands and acquires bases oftexture components (and identification information) from the server 141.

In particular, in the server 141, bases of various kinds of texturecomponents are suitably uploaded, and the updating unit 121 downloads abasis of a predetermined texture component from the server 141 asoccasion demands to acquire the basis of the predetermined texturecomponent.

Furthermore, the updating unit 121 registers the basis of the texturecomponent acquired from the server 141 into the texture DB 51 to updatethe DB data of the texture DB 51.

Here, the updating unit 121 can determine a basis of a texture to beacquired from the server 141 as occasion demands.

In particular, the updating unit 121 can acquire a basis of a texturecomponent with which the quality of an original image as an input imageto be encoded by the encoding apparatus 50 can be maintained (which isappropriate to maintain the quality) in response to the quality of theoriginal image as the input image, particularly, for example, inresponse to the S/N of the image, the resolution (whether the image is aSD (Standard Definition) image or a HD (High Definition) image or thelike), the frequency band or the like.

Further, for example, in the case where a basis of a new texturecomponent such as a basis that is new in version although it is a basisof a texture of an existing kind (for example, a basis of a texturecomponent having a higher representation effect), a basis of a texturecomponent of a new kind or the like is uploaded to the server 141, theupdating unit 121 can acquire the basis of the new texture component.

It is to be noted that the updating unit 121 not only can download(acquire) a basis of a texture component from the server 141 but alsocan upload a basis of a texture component registered in the texture DB51 to the server 141 as occasion demands.

The updating unit 131 refers to the bases of texture components as theDB data registered in the texture DB 51 through the updating unit 121 ofthe encoding apparatus 50 to determine whether or not a basis of anunregistered component that is not registered in the texture DB 63 ofthe decoding apparatus 60 exists in the bases of the texture componentsregistered in the texture DB 51.

In the case where a basis of an unregistered component exists, theupdating unit 131 accesses an external server 142 such as a server onthe Internet or the like. Then, the updating unit 131 requests theserver 142 and acquires the basis of an unregistered component and thenregisters the basis of the unregistered component into the texture DB 63to update such that the DB data of the texture DB 51 are included intothe DB data of the texture DB 63.

Here, the servers 141 and 142 are in a synchronized state with eachother and accordingly have same bases of texture components.

Further, while, in FIG. 17, the updating unit 121 accesses the server141 and the updating unit 131 accesses the server 142, the updating unit121 can access any one of the servers 141 and 142. This similarlyapplies also to the updating unit 131.

Further, while, in FIG. 17, the server 141 to be accessed by theupdating unit 121 and the server 142 to be accessed by the updating unit131 are prepared separately from each other, as the server 141 to beaccessed by the updating unit 121 and the server 142 to be accessed bythe updating unit 131, for example, a same server such as a server onthe cloud or the like can be adopted.

It is to be noted that updating of DB data of the texture DB 51 by theupdating unit 121 of the encoding apparatus 50 is not essentiallyrequired. However, in the case where updating of DB data of the textureDB 51 is to be performed, for example, by performing, for each stream ofan original image as an input image, updating of switching the basis ofa texture component to be registered into the texture DB 51, even in thecase where the capacity of the texture DB 51 is restricted to somedegree, an input image can be processed using a basis of a texturecomponent suitable for the input image.

Further, updating of DB data of the texture DB 51 by the updating unit121 of the encoding apparatus 50 or updating of DB data of the textureDB 63 by the updating unit 131 of the decoding apparatus 60 can beperformed at an arbitrary timing, for example, similarly as in the caseof the updating unit 111 of FIG. 15.

Ninth Example of Configuration of Codec

FIG. 18 is a block diagram depicting a ninth example of a configurationof a codec that restores a texture lost upon encoding.

In particular, FIG. 18 depicts an example of a configuration of a codecin which the basis of a texture component in the texture DBs 51 and 63is possible.

It is to be noted that, in FIG. 18, elements corresponding to those ofthe case of FIG. 15 are denoted by same reference numerals, anddescription of them is hereinafter omitted suitably.

Referring to FIG. 18, the codec is common to that in the case of FIG. 15in that it includes an encoding apparatus 50 and a decoding apparatus60.

Further, in FIG. 18, the encoding apparatus 50 is common to that in thecase of FIG. 15 in that it includes elements from a texture DB 51 to atransmission unit 57 and a data transmission unit 101, and the decodingapparatus 60 is common to that in the case of FIG. 15 in that itincludes elements from a reception unit 61 to a synthesis unit 66 and anupdating unit 111.

However, in FIG. 18, the encoding apparatus 50 is different from that ofFIG. 15 in that it newly includes a registration unit 151.

The registration unit 151 registers a basis of a texture componentappropriate for encoding an input image into the texture DB 51. It is tobe noted that, although necessary information is supplied from otherblocks to the registration unit 151, connection lines for supplyinginformation to the registration unit 151 are omitted in order to preventthe illustrating from being complicated.

Tenth Example of Configuration of Codec

FIG. 19 is a block diagram depicting a tenth example of a configurationof a codec that restores a texture lost upon encoding.

In particular, FIG. 19 depicts an example of a configuration of a codecin which updating of the basis of a texture component in the texture DBs51 and 63 is possible.

It is to be noted that, in FIG. 19, elements corresponding to those inthe case of FIG. 17 or 18 are denoted by same reference numerals, anddescription of them is hereinafter omitted suitably.

Referring to FIG. 19, the codec is common to that of FIG. 17 in that itincludes an encoding apparatus 50 and a decoding apparatus 60.

Further, in FIG. 19, the encoding apparatus 50 is common to that in thecase of FIG. 17 in that it includes elements from a texture DB 51 to atransmission unit 57 and an updating unit 121, and the decodingapparatus 60 is common to that of FIG. 17 in that it includes elementsfrom a reception unit 61 to a synthesis unit 66 and an updating unit131.

However, in FIG. 19, the encoding apparatus 50 is different from that ofFIG. 17 in that it newly includes the registration unit 151 of FIG. 18.

<Example of Configuration of Registration Unit 151>

FIG. 20 is a block diagram depicting an example of a configuration ofthe registration unit 151 of FIGS. 18 and 19.

Referring to FIG. 20, the registration unit 151 includes a basislearning unit 161 and a registration decision unit 162.

To the basis learning unit 161, an input image is supplied and a lowfrequency component of the input image is supplied from the separationunit 52.

The basis learning unit 161 uses, for example, an input image and a lowfrequency component of the input image as a pair of a high resolutionimage and a lower resolution image described hereinabove with referenceto FIG. 7 to perform basis learning to generate, in regard to a textureincluded in the input image, a basis of a texture component (forexample, a pair of a high resolution basis and a low resolution basisdescribed hereinabove with reference to FIG. 7).

Then, the basis learning unit 161 temporarily registers the basis of thetexture component generated by the basis learning as a basis of a newtexture component into the texture DB 51 together with identificationinformation for identifying the new texture components.

It is to be noted that the basis learning by the basis learning unit 161and temporary registration of the basis (and identification information)of the new texture component obtained by the basis learning into thetexture DB 51 can be performed at arbitrary timings.

In particular, the basis learning by the basis learning unit 161 and thetemporary registration of the basis of the new texture component can beperformed, for example, every time a frame of an input image is suppliedto the encoding apparatus 50, for each frame.

Further, the basis learning by the basis learning unit 161 and thetemporary registration of the basis of the new texture component can beperformed when the error of a match component in the case where, forexample, the basis of the new texture component is not registered in thetexture DB 51 from the input image is equal to or greater than athreshold value.

The error of the match component from the input image can be calculated,for example, in a unit of a frame. Furthermore, as the error of thematch component from the input image, a difference between pixel valuesof the match component and the input image, a difference betweenpredetermined feature amounts such as activity or the like between thematch component and the input image or the like can be adopted.

To the registration decision unit 162, an error of a restorationcomponent when, in regard to each texture component whose basis isregistered in the texture DB 51, a texture of (a noticed block of) theinput image generated from the basis of the texture component from theinput image is supplied from the match component determination unit 54.

The registration decision unit 162 performs registration decisionregarding whether or not the basis of the new texture component is to bedefinitively registered using the error of the restoration component orthe like from the match component determination unit 54.

Then, in the case where the registration decision unit 162 decides thatthe basis of the new texture component is to be definitively registered,it definitively registers the basis of the new texture componenttemporarily registered in the texture DB 51 into the texture DB 51.

In particular, the registration decision unit 162 uses the error of therestoration component from the match component determination unit 54 torecognize whether or not the error of the restoration componentgenerated from the basis of the new texture component is a minimum erroramong errors of the restoration components generated from the bases ofthe new texture components registered in the texture DB 51.

Then, in the case where the error of the restoration component generatedfrom the basis of the new texture component is a minimum error, namely,in the case where the new texture component is a match component, theregistration decision unit 162 decides whether or not a definitiveregistration condition as a predetermined condition determined inadvance to perform definitive registration is satisfied. Then, in thecase where the definitive registration condition is satisfied, theregistration decision unit 162 decides that the basis of the new texturecomponent is to be definitively registered and definitively registersthe basis of the new texture component.

As the definitive registration condition, for example, that the S/Nratio of the match component (new texture component) in the case wherethe basis of the new texture component is registered in the texture DB51 to the input image is superior by a fixed value or more to the S/N ofthe match component in the case where the basis of the new texturecomponent is not registered in the texture DB 51 to the input image orthe like can be adopted.

Further, as the definitive registration condition, for example, that anRD (Rate-Distortion) curve in the case where the basis of the newtexture component is registered in the texture DB 51 is superior by afixed value or more to an RD curve in the case where the basis of thenew texture component is not registered in the texture DB 51 or the likecan be adopted.

It is to be noted that the registration decision unit 162 candefinitively register the basis of the new texture component, forexample, in the case where, irrespective of the definitive registrationcondition, the error of the restoration component generated from thebasis of the new texture component is a minimum error among errors ofrestoration components generated from the bases of the texturecomponents registered in the texture DB 51.

Furthermore, in the case where the definitive registration condition issatisfied, the registration decision unit 162 can definitively registerthe basis of the new texture component even if the error of theregistration condition generated from the basis of the new texturecomponent is not a minimum error among the errors of the restorationcomponents generated from the bases of the texture components registeredin the texture DB 51.

Here, as a method for definitive registration of the basis of the newtexture component, for example, a method of adding and a method ofoverwriting (changing) the basis of the new texture component to (on)the texture DB 51 are available.

In the addition of the basis of the new texture component, the basis ofthe new texture component is registered in such a form that it is addedto the bases of the texture components registered in the texture DB 51.

In the overwriting of the basis of the new texture component, the basisof the new texture component is registered in such a form that it isoverwritten on the basis of some texture component registered in thetexture DB 51. In the overwriting of the basis of the new texturecomponent, the basis of the new texture component can be overwritten,for example, on the basis of a texture component that has not beendetermined as a match component, the basis of a texture component whosetiming at which it is determined as a match component is in the mostpast or a like basis from among the bases of the texture componentsregistered in the texture DB 51.

With the overwriting of the basis of the new texture component, thecapacity of the texture DB 51 can be saved.

It is to be noted that the case of addition and the case of overwritingof the basis of the new texture component are different in data amountof the bases of the texture components registered in the texture DB 51.Further, if the data amount differs, then the RD curve differs.Therefore, in the case where it is adopted as the definitiveregistration condition that the RD curve in the case where the basis ofthe new texture component is registered in the texture DB 51 is superiorby a fixed value or more to the RD curve in the case where the basis ofthe new texture component is not registered in the texture DB 51, if RDcurves in the case of addition and in the case of overwriting of thebasis of the new texture component are determined, then it can bedetermined in response to the RD curves whether the basis of the newtexture component is to be added or to be overwritten.

FIG. 21 is a flow chart illustrating an example of an encoding processof the encoding apparatus 50 of FIGS. 18 and 19.

It is to be noted that, in FIG. 21, basis learning by the basis learningunit 161 and temporary registration of a basis of a new texturecomponent are performed for each frame every time a frame of an inputimage is supplied to the encoding apparatus 50.

The encoding apparatus 50 performs an encoding process in accordancewith the flow chart of FIG. 21 successively taking each of frames of aninput image supplied to the encoding apparatus 50 as a noticed frame.

In particular, at step S151, the separation unit 52 separates a lowfrequency component from a noticed frame of an input image and suppliesthe low frequency component to the basis synthesis unit 53 and theregistration unit 151.

The basis learning unit 161 of the registration unit 151 (FIG. 20)divides the noticed frame of the input image into learning blocks asunits for which basis learning is performed, and the processing advancesfrom step S151 to step S152.

At step S152, the basis learning unit 161 selects one learning blockthat has not been selected as noticed learning block as yet as a noticedlearning block from among the learning blocks of the noticed frame ofthe input image, and the processing advances to step S153.

At step S153, the basis learning unit 161 performs basis learning forconverting a texture of the noticed learning block into a basis texture,and the processing advances to step S154.

In particular, the basis learning unit 161 uses a pair of the noticedlearning block and a block at a same position as that of the noticedlearning block within the low frequency component of the input imagefrom the separation unit 52 as the pair of a high resolution image and alow resolution image described hereinabove with reference to FIG. 7 toperform basis learning to generate of bases of a texture component ofthe noticed learning block (for example, a pair of a high resolutionbasis and a low resolution basis described hereinabove with reference toFIG. 7).

At step S154, the basis learning unit 161 determines the basis of thetexture component of the noticed learning block obtained by the basislearning as the basis of the new texture component and registers thebasis (and identification information) of the new texture component intothe texture DB 51.

Thereafter, the basis synthesis unit 53 divides the noticed frame of theinput image into blocks for defining a match component. Then, theprocessing advances from step S154 to step S155, at which the basissynthesis unit 53 selects, from among the blocks of the noticed frame ofthe input image, one block that has not been selected as the noticedblock as yet as a noticed block, and the processing advances to stepS156.

At step S156, the basis synthesis unit 53 selects, from a plurality (ofkinds) of texture components whose basis is registered (includingtemporary registration) in the texture DB 51, one texture component thathas not been selected as a noticed component as yet as a noticedcomponent, and the processing advances to step S157.

At step S157, the basis synthesis unit 53 performs, for example, basissynthesis described hereinabove with reference to FIG. 8 using the lowfrequency component of the noticed block from among the low frequencycomponents of the input image from the separation unit 52 and the basisof the noticed component to determine a restoration component thatrestores the texture component of the noticed block.

The basis synthesis unit 53 supplies the restoration component of thenoticed block to the match component determination unit 54, and theprocessing advances from step S157 to step S158.

At step S158, the match component determination unit 54 calculates anerror of the restriction component of the noticed block with respect tothe noticed block of the input image, and the processing advances tostep S159.

At step S159, the match component determination unit 54 decides whetheror not the error of the restoration component of the noticed block issmaller than a minimum error regarding the noticed block.

Here, the minimum error regarding the noticed block is a minimum valueamong errors of the restoration component of the noticed blockdetermined with regard to texture components selected as a noticedcomponent up to now from among the texture components whose basis isregistered in the texture DB 51 similarly as in the case describedhereinabove with reference to FIG. 9, and a predetermined high value isadopted as an initial value of the minimum error for a noticed block isadopted.

In the case where it is decided at step S159 that the error of therestoration component of the noticed block is not smaller than theminimum error regarding the noticed block, the processing skips stepS160 and advances to step S161.

On the other hand, in the case where it is decided at step S159 that theerror of the restoration component of the noticed block is smaller thanthe minimum error regarding the noticed block, the processing advancesto step S160.

At step S160, the match component determination unit 54 updates theminimum error regarding the noticed block to the error of therestoration component of the noticed block, namely, to the latest error,and the processing advances to step S161.

At step S161, the match component determination unit 54 decides whetheror not the error of the restoration component of the noticed block hasbeen acquired in regard to all texture components whose basis isregistered in the texture DB 51.

In the case where it is decided at step S161 that the error of therestoration component of a noticed block has not yet been acquired inregard to all texture components whose basis is registered in thetexture DB 51, the processing advances to step S156.

In other words, in the case where a texture component that has not beenselected as a noticed component as yet exists among the texturecomponents whose basis is registered in the texture DB 51, theprocessing returns from step S161 to step S156, whereafter the processesdescribed above are repeated.

On the other hand, in the case where it is decided at step S161 that theerror of the restoration component of the noticed block has beenacquired in regard to all texture components whose basis is registeredin the texture DB 51, the processing advances to step S162.

In other words, in the case where the error of the restoration componentof the noticed block has been determined in regard to all of the texturecomponents whose basis is registered in the texture DB 51, theprocessing advances from step S161 to step S162.

At step S162, the registration decision unit 162 of the registrationunit 151 (FIG. 20) performs registration decision regarding whether ornot the basis of the new texture component is to be definitivelyregistered.

In the case where it is decided at step S162 that the basis of the newtexture component is not to be definitively registered, the processingskips step S163 and advances to step S164.

In particular, for example, in the case where, in regard to the texturecomponents whose basis is registered in the texture DB 51, the error ofthe restoration component obtained from the basis of the new texturecomponent among errors of the restoration component of the noticed blockobtained by the match component determination unit 54 is not a minimumerror or, even if the error is a minimum error, a definitiveregistration condition is not satisfied, it is decided that the basis ofthe new texture component is not to be definitively registered, anddefinitive registration of the basis of the new texture component is notperformed.

On the other hand, in the case where it is decided at step S162 that thebasis of the new texture component is to be definitively registered, theprocessing advances to step S163. At step S163, the registrationdecision unit 162 definitively registers the basis of the new texturecomponent temporarily registered in the texture DB 51 into the textureDB 51, and the processing advances to step S164.

In particular, in the case where, in regard to the texture componentswhose basis is registered in the texture DB 51, the error of therestoration component obtained from the basis of the new texturecomponent among the errors of the restoration component of the noticedblock obtained by the match component determination unit 54 is a minimumerror and besides the definitive registration condition is satisfied, itis decided that the basis of the new texture component is to beregistered, and definitive registration of the basis of the new texturecomponent is performed.

At step S164, the basis synthesis unit 53 decides whether or not allblocks of the noticed frame of the input image have been selected as anoticed block.

In the case where it is decided at step S164 that all blocks of thenoticed frame of the input image have not yet been selected as a noticedblock, the processing returns to step S155, and thereafter, similarprocesses are repeated.

On the other hand, in the case where it is decided at step S164 that allblocks of the noticed frame of the input image have been selected as anoticed block, the processing advances to step S165.

At step S165, the basis learning unit 161 of the registration unit 151(FIG. 20) decides whether or not all learning blocks of the noticedframe of the input image have been selected as a noticed learning block.

In the case where it is decided at step S165 that all learning blocks ofthe noticed frame of the input image have not yet been selected as anoticed learning block, the processing returns to step S152, andthereafter, similar processes are repeated.

On the other hand, in the case where it is decided at step S165 that alllearning blocks of the noticed frame of the input image have beenselected as a noticed learning block, the processing advances to stepS166.

At step S166, the match component determination unit 54 determines, inregard to each block of the noticed frame of the input image, a texturecomponent in regard to which a minimum error is obtained among texturecomponents as restoration components of the block as a match component.Furthermore, the match component determination unit 54 acquiresidentification information of the match component in regard to eachblock of the noticed frame of the input image, and the processingadvances from step S66 to step S167.

At step S167, the transmission unit 57 generates an identificationinformation map that associates the blocks of the noticed frame of theinput image and the identification information of match components ofthe blocks (identification information from the match componentdetermination unit 54) with each other, and the processing advances tostep S168.

It is to be noted that the registration decision unit 162 of theregistration unit 151 (FIG. 20) deletes the basis (and theidentification) of the temporarily registered texture component, whichis not registered definitively in the texture DB 51 and but istemporarily registered at this point of time, from the texture DB 51.

At step S168, the removal unit 55 generates a low frequency component(of the noticed frame) of the input image obtained by removing the matchcomponent of the blocks from the basis synthesis unit 53 from the blocksof the noticed frame of the input image, namely, the difference betweenthe input image and the texture component as the match component as anencoding target image and supplies the generated encoding target imageto the encoding unit 56, and the processing advances to step S169.

At step S169, the encoding unit 56 encodes the encoding target imagefrom the removal unit 55 by a non-reversible encoding method andsupplies encoded obtained as a result of the encoding to thetransmission unit 57, and the processing advances to step S170.

At step S170, the transmission unit 57 transmits the identificationinformation map and the encoded data from the encoding unit 56, and theencoding apparatus 50 ends the processing for the noticed frame of theinput image.

It is to be noted that, since the decoding process of the decodingapparatus 60 of FIGS. 18 and 19 is similar to the decoding processdescribed with reference to FIG. 10, description of the same is omitted.

In the codec described above, the information amount of identificationinformation to be transmitted from the encoding apparatus 50 (or 30) tothe decoding apparatus 60 (or 40) not only can be controlled to a fixedvalue but also can be controlled to a variable value. Not only in thecase where the information amount of identification information iscontrolled to a fixed value but also in the case where the informationamount of identification information is controlled to a variable value,as the information amount of identification information increases, thetransmission efficiency degrades by an amount corresponding to theincreasing information amount. However, it is possible to improve thepicture quality of a decoded image (output image) as much.

Here, a block of a target whose match component is to be determined inan input image or a block of a target whose match component is to besynthesized in a decoded image is referred to as target block. Further,a block (region) of a texture component to be generated by basissynthesis is referred to as texture block.

Further, as the information amount of identification information, forexample, a data amount of identification information to be transmittedwith respect to an input image of one frame (picture) is adopted. Inthis case, the information amount of identification information isrepresented by the bit number of one piece of identificationinformation×number of target blocks that configure the input image ofone frame.

It is to be noted that, although all of the target blocks that configurethe input image of one frame need not have an equal size, in order tosimplify the description, it is assumed here that all of the targetblocks that configure the input image of one frame have an equal size.

For example, in the case where one frame of the input image is a HD(High Definition) image of 1920×1080 pixels and a block of 192×108pixels obtained by dividing the image into 10 in both of the horizontaland vertical directions is a target block, one frame is configured from100 target blocks. In this case, the number of pieces of identificationinformation of one frame is 100 equal to the number of target blocks.Further, for example, in the case where one frame is configured from1000 target blocks, the number of pieces of identification informationof one frame is 1000 equal to the number of target blocks.

It is to be noted that, as the number of target blocks that configureone frame increases, the size of the target blocks decreases. Although,in the present embodiment, though not depicted, it is presupposed thatthe size of a target block and the size of a texture block generated bybasis synthesis are equal to each other, the sizes of a target block anda texture block may not be equal to each other. In other words, as atarget block, a block of a size equal to or smaller than that of atexture block can be adopted. In the case where a target block issmaller than a texture block, for example, the texture at a portion suchas a central portion or the like of a texture block can be adopted as atexture of a target block.

As the size of a target block decreases, the picture quality of arestoration image (output image) obtained by synthesis of a decodedimage and a texture component as a match component generally improves.

It is to be noted that the size of a target block is restricted to asize equal to or smaller than the size of a texture block. Accordingly,the input image cannot be divided into target blocks of a size exceedingthe size of a texture block, and the number of target blocks when aninput image is divided into target blocks is restricted by the size of atexture block.

In the case where the information amount of identification informationis controlled to a fixed value in the codec, the bit number of one pieceof identification information and the number of target blocks of oneframe are individually controlled to fixed values. Alternatively, thebit number of one piece of identification information and the number oftarget blocks of one frame are individually controlled to variablevalues such that the information amount of identificationinformation=bit number of one piece of identification information×numberof target blocks of one frame becomes a fixed value.

It is to be noted that, in the case where (bases of) 2N texturecomponents are stored in the texture DB 51 (similarly also with regardto the texture DBs 31, 43 and 63), the 2N texture components can beidentified by identification information of N bits. In this case, forexample, by associating the 2N texture components stored in the textureDB 51 with the 2N pieces of identification information represented by Nbits and by associating the 2N−1 texture components from among the 2Ntexture components stored in the texture DB 51 with the 2N−1 pieces ofidentification information represented, for example, by N−1 bits smallerthan N bits, the number of bits of identification information can becontrolled to N bits or N−1 bits.

Further, the number of target blocks of one frame can be controlledwithin a range within which a target block becomes a block of a sizeequal to or smaller than the size of a texture block.

In the case where the information amount of identification informationis controlled to a variable value in the codec, one or both of the bitnumber of one piece of identification information and the number oftarget blocks of one frame are adaptively controlled to a variablevalue.

Here, in the case where the information amount of identificationinformation is controlled to a variable value, it is possible toadaptively control the information amount of identification informationin response to the picture quality (bitrate) of an input mage, the size(pixel number) of one frame, the genre (for example, sports, animation,movie and so forth) and so forth.

Further, in the codec, in addition to identification information,additional information that can be used for generation of a texturecomponent that contributes to improvement of the picture quality of adecoded image can be generated by the encoding apparatus 50 (or 30) andtransmitted to the decoding apparatus 60 (or 40).

As (the type of) the additional information, for example, gaininformation that determines the amplitude of a texture component as amatch component, a parameter that can be used for generation of atexture component, an image feature amount of an input image and soforth are available.

The gain information as the additional information is used for controlof the gain of a texture component obtained by basis synthesis. Inparticular, in the basis synthesis, a texture component whose amplitudeis normalized is determined as occasion demands. The gain informationcan be used for determination of an amplitude of a texture componentwhose amplitude is normalized.

As the parameter as the additional information, for example, bandinformation of a texture of an input image (a result of FFT (FastFourier Transform) of a texture of an input image or the like) and soforth are available.

For example, in the case where the degree to which a texture componentas a match component that best matches with (a target block of) an inputimage among texture components stored in the texture DB 51 (and thetexture DBs 31, 43 and 63) matches with the input image is equal to orlower than a threshold value, the encoding apparatus 50 (or 30) cantransmit the band information as the additional information to thedecoding apparatus 60 (or 40).

In this case, the decoding apparatus 60 can improve the picture qualityof a decoded image by filtering a texture component as a match componentsuch that it has a band similar to the band represented by the bandinformation as the additional information and synthesizing the texturecomponent after the filtering with the decoded image.

Further, as the image feature amount as the additional information, a DR(dynamic range) of pixel values in a predetermined region such as oneframe, a target block or the like of the input image, the variance ofthe DR, an adjacent pixel difference of the input image, luminanceinformation of the input image and so forth are available.

For example, in the case where the degree to which the texture componentas a match component that best matches with the input image among thetexture components stored in the texture DB 51 matches with the inputimage is equal to or lower than the threshold value, the encodingapparatus 50 can transmit the image feature amount as the additionalinformation to the decoding apparatus 60.

In this case, the decoding apparatus 60 can improve the picture qualityof a decoded image by processing the texture component as the matchcomponent such that it becomes a texture component of the image featureamount similar to the image feature amount as the additional informationand synthesizing the texture component after the processing with thedecoded image.

In the codec, the number of pieces of additional information to betransmitted from the encoding apparatus 50 to the decoding apparatus 60can be increased or decreased, for example, depending upon suchconditions as a processing performance of the codec (or an incorporatingapparatus in which the codec is incorporated), the operation cost (powerconsumption, heat generation and so forth) and so forth.

For example, in the case where the processing performance of the codecis high, namely, in the case where it is permitted that the number oftimes of arithmetic operations necessary for processing per one pixel isgreat (number of times of arithmetic operation per unit time period isgreat), in the case where the power versus operating cost is superior orin a like case, it is possible to additionally transmit, in addition tothe identification information, a parameter or an image feature amountas the additional information to improve the picture quality of adecoded image.

Further, for example, in the case where the processing performance ofthe codec is low, it is possible to transmit gain information as theadditional information in a unit of one pixel from the encodingapparatus 50 to the decoding apparatus 60. In this case, in the decodingapparatus 60, the gain information in a unit of one pixel can be used asit is for control of the amplitude of the texture component.

Furthermore, in the case where the processing performance of the codecis high, it is possible to reduce the transmission amount of theadditional information by transmitting gain information as theadditional information in a unit of a plurality of pixels from theencoding apparatus 50 to the decoding apparatus 60. In this case, in thedecoding apparatus 60, the gain information of a unit of a plurality ofpixels can be used for control of the amplitude of a texture componentby performing interpolation such that the gain information becomes gaininformation of a unit of one pixel.

It is to be noted that, in a conventional non-reversible encoding methodsuch as HEVC or the like, a texture of an image is subjected to DCT(Discrete Cosine Transform) and is quantized further, whereafter it istransmitted. In this case, in order to improve the picture quality ofthe texture, it is necessary to increase the transmission rate andperform quantization with a fine quantization step.

On the other hand, in the encoding apparatus 50 (or 30), a texture istransmitted by identification information. In this manner, the encodingapparatus 50 is different from that of a conventional non-reversibleencoding method in that a texture is transmitted by identificationinformation in the encoding apparatus 50.

Here, the applicant of the present application has proposed aclassification adaptive process previously.

In the classification adaptive process, for example, a first image isconverted into a second image. In such a classification adaptiveprocess, a pixel that becomes a prediction tap to be used for predictionarithmetic operation for determining a pixel value of a correspondingpixel of the second image corresponding to a noticed pixel noticed inthe first image is selected from within the first image, and the noticedpixel is classified into one of a plurality of classes in accordancewith a fixed rule. Then, in the classification adaptive process, fromtap coefficients to be used for prediction arithmetic operation for aplurality of classes determined by learning that minimizes the sum totalof square errors as statistical errors between a result of predictionarithmetic operation in which a child image corresponding to the firstimage is used and a teacher image corresponding to the second image, thetap coefficient of the class of the noticed pixel is acquired, and apixel value of a corresponding pixel is determined by performingprediction arithmetic operation in which the tap coefficient of theclass of the noticed pixel and the prediction tap of the noticed pixelare used.

In the classification adaptive process, in learning of a tapcoefficient, a tap coefficient is determined using it as a norm fordetermination of a tap coefficient that the sum total of square errorsof a result of prediction arithmetic operation in which a child imagecorresponding to the first image is used and a teacher imagecorresponding to the second image is minimized.

Also in the encoding apparatus 50 (or 30), as a match componentdetermination norm for determining a match component that matches withan input image among texture components stored in the texture DB 51 (or31), it can be adopted that the sum total of square errors between aninput image and a texture component is minimized (hereinafter referredto as square error minimum norm) similarly as in the classificationadaptive process.

Furthermore, the encoding apparatus 50 can control the picture qualityor the subjective performance (mainly of a texture) of a decoded imageby adopting a norm other than the square error minimum norm as a matchcomponent determination norm.

As the match component determination norm other than the square errorminimum norm, for example, SSIM (Structural Similarity) that is an indexproximate to qualitative or the like can be used.

The present technology is different from the classification adaptiveprocess that adopts the square error minimum norm in that it is possibleto adopt a norm other than the square error minimum norm as the matchcomponent determination norm and control the picture quality or thesubjective performance of a decoded image in such a manner as describedabove.

Here, the subjective performance signifies a performance that an imagecharacteristic such as, for example, refinement, sharpness, a sense ofresolution, a sense of contrast and so forth acts on qualitativerecognition or impression of an evaluator who evaluates the pictureequality.

A match component determination norm by which, from an input image of acertain particular subjective performance or picture quality, a decodedimage of a desired subjective performance or picture quality (forexample, a decoded image from which more refinement or sharpness can besensed or the like) is obtained can be created (designed) by repeating,for an input image of a particular subjective performance or picturequality, a qualitative evaluation examination performed by an adjuster,a user or the like of the codec.

Upon creation of such a match component determination norm by which adecoded image of a desired subjective performance or picture quality isobtained as described above, it is necessary that (a subjectiveperformance or picture quality of) an input image to be used for suchcreation is known.

<Application to Multi-View Image Encoding and Decoding System>

FIG. 22 is a view depicting an example of a multi-view image encodingmethod.

As depicted in FIG. 22, a multi-view image includes images of aplurality of viewpoints (views (view)). A plurality of views ofmulti-view images include a base view whose encoding and decoding areperformed using only an image of the own view without using informationof any other view and a non-base view whose encoding and decoding areperformed using information of some other view. Encoding and decoding ofa non-base view may be performed using information of a base view or maybe performed using information of a different non-base view.

In the case where a multi-view image as in the example of FIG. 22 is tobe encoded and decoded, the multi-view image is encoded for eachviewpoint. Then, in the case where encoded data obtained in this mannerare to be decoded, encoded data of each viewpoint are individuallydecoded (namely, for each viewpoint). To such encoding and decoding ofeach viewpoint, the method described hereinabove in the foregoingdescription of the embodiment may be applied. By such application, thetransmission efficiency and the picture quality can be improved. Inshort, also in the case of a multi-view image, the transmissionefficiency and the picture quality can be improved.

<Multi-View Image Encoding and Decoding System>

FIG. 23 is a view depicting a multi-view image encoding apparatus of amulti-view image encoding and decoding system that performs multi-viewimage encoding and decoding described hereinabove.

As depicted in FIG. 23, the multi-view image encoding apparatus 1000includes an encoding unit 1001, another encoding unit 1002 and amultiplexing unit 1003.

The encoding unit 1001 encodes a base view image to generate a base viewimage encoded stream. The encoding unit 1002 encodes a non-base viewimage to generate a non-base view image encoded stream. The multiplexingunit 1003 multiplexes the base view image encoded stream generated bythe encoding unit 1001 and the non-base view image encoded streamgenerated by the encoding unit 1002 to generate a multi-view imageencoded stream.

FIG. 24 is a view depicting a multi-view image decoding apparatus thatperforms multi-view image decoding described hereinabove.

As depicted in FIG. 24, the multi-view image decoding apparatus 1010includes a demultiplexing unit 1011, a decoding unit 1012 and anotherdecoding unit 1013.

The demultiplexing unit 1011 demultiplexes a multi-view image encodedstream in which a base view image encoded stream and a non-base viewimage encoded stream are multiplexed to extract the base view imageencoded stream and the non-base view image encoded stream. The decodingunit 1012 decodes the base view image encoded stream extracted by thedemultiplexing unit 1011 to obtain a base view image. The decoding unit1013 decodes the non-base view image encoded stream extracted by thedemultiplexing unit 1011 to obtain a non-base view image.

For example, in such a multi-view image encoding and decoding system asdescribed above, the encoding apparatus 10 described hereinabove in theforegoing description of the embodiment may be applied as the encodingunit 1001 and the encoding unit 1002 of the multi-view image encodingapparatus 1000. By this application, also in encoding of a multi-viewimage, the method described in the foregoing description of theembodiment can be applied. In other words, it is possible to improve thetransmission efficiency and the picture quality. Further, for example,as the decoding unit 1012 and the decoding unit 1013 of the multi-viewimage decoding apparatus 1010, the decoding apparatus 20 described inthe foregoing description of the embodiment may be applied. By thisapplication, also in decoding of encoded data of a multi-view image, themethod described in the foregoing description of the embodiment can beapplied. In other words, it is possible to improve the transmissionefficiency and the picture quality.

<Application to Hierarchical Image Encoding and Decoding System>

Further, the series of processes described above can be applied to ahierarchical image encoding (scalable encoding) and decoding system.

FIG. 25 is a view depicting an example of a hierarchical image encodingmethod.

Hierarchical image encoding (scalable encoding) converts image data intoa plurality of layers (hierarchies) of images so as to have ascalability (scalability) function in regard to a predeterminedparameter and encodes the image data for each layer. Hierarchical imagedecoding (scalable decoding) is decoding corresponding to thehierarchical image encoding.

As depicted in FIG. 25, in hierarchization of an image, one image isdivided into a plurality of images (layers) with reference to apredetermined parameter having a scalability function. In short, ahierarchized image (hierarchical image) includes a plurality of imagesof different hierarchies (layers) among which the value of thepredetermined parameter is different. The plurality of layers of thehierarchical image include a base layer that allows encoding anddecoding using only an image of its own layer without utilizing an imageof any other layer and a non-base layer (referred to also as enhancementlayer) that allows encoding and decoding using an image of a differentlayer. The non-base layer may utilize an image of the base layer or mayutilize a different image of the non-base layer.

Generally, a non-base layer is configured from data of a differenceimage (difference data) between an own image of the non-base layer andan image of a different layer such that the redundancy may be reduced.For example, in the case where one image is hierarchized into two layersof a base layer and a non-base layer (also called enhancement layer), animage of lower quality than that of the original image is obtained fromonly data of the base layer, and by synthesizing data of the base layerand data of the non-base layer, the original image (namely, the image ofhigh quality) is obtained.

By hierarchizing an image in this manner, images of various qualitiescan be obtained readily according to the situation. For example, to aterminal having a low processing capacity like a portable telephone set,image compression information only of the base layer (base layer) istransmitted, and a moving image that has a low space time resolution oris not high in picture quality is reproduced. However, to a terminalhaving a high processing capacity like a television set or a personalcomputer, image compression information of the enhancement layer(enhancement layer) is transmitted in addition to the base layer (baselayer), and a moving image that has a high space time resolution or ishigh in picture quality is reproduced. In this manner, image compressioninformation according to the capacity of a terminal or network can betransmitted from a server without performing a transcode process.

In the case where such a hierarchical image as in the example of FIG. 25is encoded and decoded, the hierarchical image is encoded for eachlayer. Then, when the encoded data obtained in this manner are to bedecoded, the encoded data of each other is individually decoded (namely,for each layer). To such encoding and decoding of each layer, the methoddescribed in the above-described embodiment may be applied. By suchapplication, the transmission efficiency and the picture quality can beimproved. In short, also in the case of a hierarchical image, thetransmission efficiency and the picture quality can be improved.

<Scalable Parameter>

In such hierarchical image encoding and hierarchical image decoding(scalable encoding and scalable decoding), a parameter having thescalability (scalability) function is arbitrary. For example, a spaceresolution may be made the parameter (spatial scalability). In the caseof this spatial scalability (spatial scalability), the resolution of animage differs for each layer.

Further, as a parameter that has such a scalability property asdescribed above, for example, a time resolution may be applied (temporalscalability). In the case of this temporal scalability (temporalscalability), the frame rate differs for each layer.

Further, as a parameter that has such a scalability property asdescribed above, for example, a signal to noise ratio (SNR (Signal toNoise ratio)) may be applied (SNR scalability). In the case of this SNRscalability (SNR scalability), the SN ratio differs for each layer.

The parameter that has such a scalability property may naturally beother than the examples described above. For example, bit depthscalability (bit-depth scalability) is available in which the base layer(base layer) is configured from an 8-bit (bit) image and a 10-bit (bit)image is obtained by adding an enhancement layer (enhancement layer) tothe 8-bit (bit) image.

Further, chroma scalability (chroma scalability) is available in whichthe base layer is configured from a component image of the 4:2:0 formatand a component image of the 4:2:2 format is obtained by adding anenhancement layer (enhancement layer) to the component image.

<Hierarchical Image Encoding and Decoding System>

FIG. 26 is a view depicting a hierarchical image encoding apparatus of ahierarchical image encoding and decoding system that performs thehierarchical image encoding and decoding described above.

As depicted in FIG. 23, the hierarchical image encoding apparatus 1020includes an encoding unit 1021, another encoding unit 1022 and amultiplexing unit 1023.

The encoding unit 1021 encodes a base layer image to generate a baselayer image encoded stream. The encoding unit 1022 encodes a non-baseimage to generate a non-base layer image encoded stream. Themultiplexing unit 1023 multiplexes the base layer image encoded streamgenerated by the encoding unit 1021 and the non-base layer image encodedstream generated by the encoding unit 1022 to generate a hierarchicalimage encoded stream.

FIG. 27 is a view depicting a hierarchical image decoding apparatus thatperforms hierarchical image decoding described hereinabove.

As depicted in FIG. 27, the hierarchical image decoding apparatus 1030includes a demultiplexing unit 1031, a demultiplexing unit 1032 andanother decoding unit 1033.

The demultiplexing unit 1031 demultiplexes a hierarchical image encodedstream in which a base layer image encoded stream and a non-base layerimage encoded stream are multiplexed to extract the base layer imageencoded stream and the non-base layer image encoded stream. Thedemultiplexing unit 1032 decodes the base layer image encoded streamextracted by the demultiplexing unit 1031 to obtain a base layer image.The decoding unit 1033 decodes the non-base layer image encoded streamextracted by the demultiplexing unit 1031 to obtain a non-base layerimage.

For example, in such a hierarchical image encoding and decoding systemas described above, the encoding apparatus 10 described hereinabove inthe foregoing description of the embodiment may be applied as theencoding unit 1021 and the encoding unit 1022 of the hierarchical imageencoding apparatus 1020. By this application, also in encoding of ahierarchical image, the method described in the foregoing description ofthe embodiment can be applied. In other words, it is possible to improvethe transmission efficiency and the picture quality. Further, forexample, as the demultiplexing unit 1032 and the decoding unit 1033 ofthe hierarchical image decoding apparatus 1030, the decoding apparatus20 described in the foregoing description of the embodiment may beapplied. By this application, also in decoding of encoded data of ahierarchical image, the method described in the foregoing description ofthe embodiment can be applied. In other words, it is possible to improvethe transmission efficiency and the picture quality.

<Computer>

It is to be noted that, while the series of processes described abovecan be executed by hardware, it may otherwise be executed by software.Where the series of processes is executed by software, a program thatconstructs the software is installed into a computer. Here, the computerincludes a computer incorporated in hardware for exclusive use, apersonal computer, for example, for universal use that can executevarious functions by installing various programs, and so forth.

FIG. 28 is a block diagram depicting an example of a configuration ofhardware of a computer that executes the series of processes describedabove in accordance with a program.

In the computer 1100 depicted in FIG. 28, a CPU (Central ProcessingUnit) 1101, a ROM (Read Only Memory) 1102 and a RAM (Random AccessMemory) 1103 are connected to each other by a bus 1104.

To the bus 1104, also an input/output interface 1110 is connected. Tothe input/output interface 1110, an inputting unit 1111, an outputtingunit 1112, a storage unit 1113, a communication unit 1114 and a drive1115 are connected.

The inputting unit 1111 is configured, for example, from a keyboard, amouse, a microphone, a touch panel, an input terminal and so forth. Theoutputting unit 1112 is configured from a display, a speaker, an outputterminal and so forth. The storage unit 1113 is configured, for example,from a hard disk, a RAM disk, a nonvolatile memory and so forth. Thecommunication unit 1114 is configured, for example, from a networkinterface. The drive 1115 drives a removable medium 821 such as amagnetic disk, an optical disk, a magneto-optical disk, a semiconductormemory or the like.

In the computer configured in such a manner as described above, the CPU1101 loads a program stored, for example, in the storage unit 1113 intothe RAM 1103 through the input/output interface 1110 and the bus 1104and executes the program to perform the series of processes describedabove. Into the RAM 1103, also data and so forth necessary uponexecution of various processes by the CPU 1101 are suitably stored.

The program executed by the computer (CPU 1101) can be recorded, forexample, into the removable medium 821 as a package medium or the likeand applied. In this case, the program can be installed into the storageunit 1113 through the input/output interface 1110 by mounting theremovable medium 821 on the drive 1115.

Further, this program can be provided through a wired or wirelesstransmission medium such as a local area network, the Internet or adigital satellite broadcast. In this case, the program can be receivedby the communication unit 1114 and installed into the storage unit 1113.

Also it is possible to install this program in advance into the ROM 1102or the storage unit 1113.

<Application of Present Technology>

The encoding apparatus 10 or the decoding apparatus 20 according to theembodiment described hereinabove can be applied to various electronicapparatus such as transmitters or receives, for example, fordistribution by a satellite broadcast, a wired broadcast such as a cableTV or the Internet, distribution to a terminal by cellular communicationand so forth, or recording apparatus that record an image on a mediumsuch as an optical disk, a magnetic disk, a flash memory and so forth,reproduction apparatus for reproducing an image from such storage mediaas described above and so forth. In the following, four examples ofapplication are described.

First Application Example: Television Receiver

FIG. 29 is a view depicting an example of a schematic configuration of atelevision apparatus to which the embodiment described hereinabove isapplied.

The television apparatus 1200 includes an antenna 1201, a tuner 1202, ademultiplexer 1203, a decoder 1204, a video signal processing unit 1205,a display unit 1206, an audio signal processing unit 1207, a speaker1208, an external interface (I/F) unit 1209, a control unit 1210, a userinterface (I/F) unit 1211, and a bus 1212.

The tuner 1202 extracts a signal of a desired channel from broadcastingsignals received through the antenna 1201 and demodulates the extractedsignal. Then, the tuner 1202 outputs an encoded bit stream obtained bythe demodulation to the demultiplexer 1203. In particular, the tuner1202 has a role as a transmission unit in the television apparatus 1200,which receives an encoded stream in which an image is encoded.

The demultiplexer 1203 demultiplexes a video stream and an audio streamof a broadcasting program of a viewing target from an encoded bit streamand outputs the demultiplexed streams to the decoder 1204. Further, thedemultiplexer 1203 extracts auxiliary data such as an EPG (ElectronicProgram Guide) or the like from the encoded bit stream and supplies theextracted data to the control unit 1210. It is to be noted that thedemultiplexer 1203 may perform descramble in the case where the encodedbit stream is in a scrambled state.

The decoder 1204 decodes the video stream and the audio stream inputtedfrom the demultiplexer 1203. Then, the decoder 1204 outputs video datagenerated by the decoding process to the video signal processing unit1205. Further, the decoder 1204 outputs audio data generated by thedecoding process to the audio signal processing unit 1207.

The video signal processing unit 1205 reproduces video data inputtedfrom the decoder 1204 and causes the display unit 1206 to display avideo. Further, the video signal processing unit 1205 may cause thedisplay unit 1206 to display an application screen image suppliedthrough the network. Further, the video signal processing unit 1205 mayperform additional processes such as, for example, noise removal and soforth for video data in accordance with a setting. Furthermore, thevideo signal processing unit 1205 may generate an image of a GUI(Graphical User Interface) such as, for example, a menu, a button, acursor or the like and cause the generated image to be superimposed onan output image.

The display unit 1206 is driven by a drive signal supplied from thevideo signal processing unit 1205 and displays a video or an image on avideo face of a display device (for example, a liquid crystal display, aplasma display, an OELD (Organic Electro Luminescence Display) (organicEL display) or the like).

The audio signal processing unit 1207 performs a reproduction processsuch as D/A conversion, amplification and so forth for audio datainputted from the decoder 1204 and causes sound to be outputted from thespeaker 1208. Further, the audio signal processing unit 1207 may performadditional processes such as noise removal or the like for the audiodata.

The external interface unit 1209 is an interface for connecting thetelevision apparatus 1200 and an external apparatus or a network to eachother. For example, a video stream or an audio stream received throughthe external interface unit 1209 may be decoded by the decoder 1204. Inparticular, also the external interface unit 1209 has a role as atransmission unit in the television apparatus 1200, which receives anencoded stream in which images are encoded.

The control unit 1210 includes a processor such as a CPU, and a memorysuch as a RAM, a ROM and so forth. The memory stores a program to beexecuted by the CPU, program data, EPG data, data acquired through anetwork and so forth. The program stored in the memory is read into andexecuted by the CPU, for example, upon activation of the televisionapparatus 1200. The CPU executes the program to control operation of thetelevision apparatus 1200 in response to an operation signal inputted,for example, from the user interface unit 1211.

The user interface unit 1211 is connected to the control unit 1210. Theuser interface unit 1211 includes a button and a switch for allowing,for example, a user to operate the television apparatus 1200, areception unit for a remote controlling signal and so forth. The userinterface unit 1211 detects an operation by the user through thecomponents mentioned and generates an operation signal, and outputs thegenerated operation signal to the control unit 1210.

The bus 1212 connects the tuner 1202, demultiplexer 1203, decoder 1204,video signal processing unit 1205, audio signal processing unit 1207,external interface unit 1209 and control unit 1210 to each other.

In the television apparatus 1200 configured in such a manner asdescribed above, the decoder 1204 may have a function of the decodingapparatus 20 described hereinabove. In particular, the decoder 1204 maydecode encoded data by a method described hereinabove in connection withthe embodiment. By such decoding, the television apparatus 1200 canimprove the transmission efficiency and the picture quality.

Further, in the television apparatus 1200 configured in such a manner asdescribed above, the video signal processing unit 1205 may beconfigured, for example, so as to encode image data supplied from thedecoder 1204 and output the obtained encoded data to the outside of thetelevision apparatus 1200 through the external interface unit 1209.Further, the video signal processing unit 1205 may have the function ofthe encoding apparatus 10 described hereinabove. In short, the videosignal processing unit 1205 may encode image data supplied from thedecoder 1204 by the method described hereinabove in connection with theembodiment. By such encoding, the television apparatus 1200 can improvethe transmission efficiency and the picture quality.

Second Application Example: Portable Telephone Set

FIG. 30 is a view depicting an example of a schematic configuration of aportable telephone set to which the embodiment described hereinabove isapplied.

The portable telephone set 1220 includes an antenna 1221, acommunication unit 1222, an audio codec 1223, a speaker 1224, amicrophone 1225, a camera unit 1226, an image processing unit 1227, ademultiplexing unit 1228, a recording and reproduction unit 1229, adisplay unit 1230, a control unit 1231, an operation unit 1232 and a bus1233.

The antenna 1221 is connected to the communication unit 1222. Thespeaker 1224 and the microphone 1225 are connected to the audio codec1223. The operation unit 1232 is connected to the control unit 1231. Thebus 1233 connects the communication unit 1222, audio codec 1223, cameraunit 1226, image processing unit 1227, demultiplexing unit 1228,recording and reproduction unit 1229, display unit 1230 and control unit1231 to each other.

The portable telephone set 1220 performs such operations as transmissionand reception of a voice signal, transmission and reception of anelectronic mail or image data, pickup of an image, recording of data andso forth in various operation modes including a voice speech mode, adata communication mode, an image pickup mode and a videophone mode.

In the voice speech mode, an analog speech signal generated by themicrophone 1225 is supplied to the audio codec 1223. The audio codec1223 converts the analog speech signal into speech data and A/D convertsand compresses the speech data after the conversion. Then, the audiocodec 1223 outputs the speech data after the compression to thecommunication unit 1222. Then, the communication unit 1222 encodes andmodulates the speech data to generate a transmission signal. Then, thecommunication unit 1222 transmits the generated transmission signal to abase station (not depicted) through the antenna 1221. On the other hand,the communication unit 1222 amplifies and frequency converts a wirelesssignal received through the antenna 1221 to acquire a reception signal.Then, the communication unit 1222 demodulates and decodes the receptionsignal to generate speech data and outputs the generated speech data tothe audio codec 1223. The audio codec 1223 decompresses and D/A convertsthe speech data to generate an analog speech signal. Then, the audiocodec 1223 supplies the generated speech signal to the speaker 1224 suchthat speech is outputted from the speaker 1224.

On the other hand, in the data communication mode, for example, thecontrol unit 1231 generates character data that configure an electronicmail in response to operations by the user through the operation unit1232. Further, the control unit 1231 controls the display unit 1230 todisplay characters. Further, the control unit 1231 generates electronicmail data in response to a transmission instruction from the userthrough the operation unit 1232 and outputs the generated electronicmail data to the communication unit 1222. The communication unit 1222encodes and modulates the generated electronic mail data to generate atransmission signal. Then, the communication unit 1222 transmits thegenerated transmission signal to the base station (not depicted) throughthe antenna 1221. On the other hand, the communication unit 1222amplifies and frequency converts a wireless signal received through theantenna 1221 to acquire a reception signal. Then, the communication unit1222 demodulates and decodes the reception signal to restore theelectronic mail data and outputs the restored electronic mail data tothe control unit 1231. The control unit 1231 controls the display unit1230 to display the substance of the electronic mail and supplies theelectronic data to the recording and reproduction unit 1229 such thatthe electronic data is written into its recording medium.

The recording and reproduction unit 1229 has an arbitrary storage mediumthat is readable and writable. For example, the storage medium may be abuilt-in type storage medium such as a RAM, a flash memory or the likeor an externally mountable storage medium such as a hard disk, amagnetic disk, a magneto-optical disk, an optical disk, a USB (UniversalSerial Bus) memory, a memory card or the like.

Further, in the image pickup mode, for example, the camera unit 1226picks up an image of an image pickup object to generate image data andoutputs the generated image data to the image processing unit 1227. Theimage processing unit 1227 encodes the image data inputted from thecamera unit 1226 and supplies the encoded stream to the recording andreproduction unit 1229 so as to be written into the storage medium ofthe same.

Further, in the image display mode, the recording and reproduction unit1229 reads out an encoded stream recorded on the storage medium andoutputs the encoded stream to the image processing unit 1227. The imageprocessing unit 1227 decodes the encoded stream inputted from therecording and reproduction unit 1229 and supplies the image data to thedisplay unit 1230 such that the image is displayed.

Further, in the videophone mode, for example, the demultiplexing unit1228 multiplexes a video stream encoded by the image processing unit1227 and an audio stream inputted from the audio codec 1223 and outputsthe multiplexed stream to the communication unit 1222. The communicationunit 1222 encodes and modulates the stream to generate a transmissionsignal. Then, the communication unit 1222 transmits the generatedtransmission signal to a base station (not depicted) through the antenna1221. On the other hand, the communication unit 1222 amplifies andfrequency converts a wireless signal received through the antenna 1221to acquire a reception signal. The transmission signal and the receptionsignal can include an encoded bit stream. Then, the communication unit1222 demodulates and decodes the reception signal to restore the streamand outputs the restored stream to the demultiplexing unit 1228. Thedemultiplexing unit 1228 demultiplexes the video stream and the audiostream from the inputted stream and outputs the video stream to theimage processing unit 1227 while it outputs the audio stream to theaudio codec 1223. The image processing unit 1227 decodes the videostream to generate video data. The video data is supplied to the displayunit 1230, by which a series of images are displayed. The audio codec1223 decompresses and D/A converts the audio stream to generate ananalog audio signal. Then, the audio codec 1223 supplies the generatedaudio signal to the speaker 1224 such that speech is outputted from thespeaker 1224.

In the portable telephone set 1220 configured in this manner, forexample, the image processing unit 1227 may have the function of theencoding apparatus 10 described hereinabove. In short, the imageprocessing unit 1227 may encode image data by the method described inthe foregoing description of the embodiment. By such encoding, theportable telephone set 1220 can improve the transmission efficiency andthe picture quality.

Further, in the portable telephone set 1220 configured in this manner,for example, the image processing unit 1227 may have the function of thedecoding apparatus 20 described hereinabove. In short, the imageprocessing unit 1227 may decode encoded data by the method describedhereinabove in the description of the embodiment. By such decoding, theportable telephone set 1220 can improve the transmission efficiency andthe picture quality.

Third Application Example: Recording and Reproduction Apparatus

FIG. 31 is a view depicting an example of a schematic configuration of arecording and reproduction apparatus to which the embodiment describedhereinabove is applied.

The recording and reproduction apparatus 1240 encodes, for example,audio data and video data of a received broadcasting program and recordsthe encoded data on a recording medium. Further, the recording andreproduction apparatus 1240 may encode, for example, audio data andvideo data acquired from a different apparatus and record the data on arecording medium. Further, the recording and reproduction apparatus 1240reproduces, for example, data recorded on the recording medium on amonitor and a speaker in response to an instruction of the user. At thistime, the recording and reproduction apparatus 1240 decodes audio dataand video data.

The recording and reproduction apparatus 1240 includes a tuner 1241, anexternal interface (I/F) unit 1242, an encoder 1243, an HDD (Hard DiskDrive) unit 1244, a disk drive 1245, a selector 1246, a decoder 1247, anOSD (On-Screen Display) unit 1248, a control unit 1249 and a userinterface (I/F) 1250.

The tuner 1241 extracts a signal of a desired channel from broadcastingsignals received through an antenna (not depicted) and demodulates theextracted signal. Then, the tuner 1241 outputs an encoded bit streamobtained by demodulation to the selector 1246. In other words, the tuner1241 has a role as the transmission unit in the recording andreproduction apparatus 1240.

The external interface unit 1242 is an interface for connecting therecording and reproduction apparatus 1240 and an external apparatus or anetwork to each other. The external interface unit 1242 may be, forexample, an IEEE (Institute of Electrical and Electronic Engineers) 1394interface, a network interface, a USB interface, a flash memoryinterface or the like. For example, video data and audio data receivedthrough the external interface unit 1242 are inputted to the encoder1243. In other words, the external interface unit 1242 has a role as thetransmission unit in the recording and reproduction apparatus 1240.

The encoder 1243 encodes, in the case where video data and audio datainputted from the external interface unit 1242 are not in an encodedstate, the video data and the audio data. Then, the encoder 1243 outputsan encoded bit stream to the selector 1246.

The HDD unit 1244 records an encoded bit stream, in which content dataof videos and audios are compressed, various programs and other datainto an internal hard disk. Further, the HDD unit 1244 reads out, uponreproduction of videos and audios, such data from the hard disk.

The disk drive 1245 performs recording and reading out of data on andfrom a recording medium mounted thereon. The recording medium to bemounted on the disk drive 1245 may be, for example, a DVD (DigitalVersatile Disc) disk (DVD-Video, DVD-RAM (DVD-Random Access Memory),DVD-R (DVD-Readable), DVD-RW (DVD-Rewritable), DVD+R (DVD+Recordable),DVD+RW (DVD+Rewriteable) and so forth) or a Blu-ray (registeredtrademark) disk or the like.

The selector 1246 selects, upon recording of videos and audios, anencoded bit stream inputted from the tuner 1241 or the encoder 1243 andoutputs the selected encoded bit stream to the HDD unit 1244 or the diskdrive 1245. On the other hand, upon reproduction of videos and audios,the selector 1246 outputs an encoded bit stream inputted from the HDDunit 1244 or the disk drive 1245 to the decoder 1247.

The decoder 1247 decodes an encoded bit stream to generate video dataand audio data. Then, the decoder 1247 outputs the generated video datato the OSD unit 1248. Further, the decoder 1247 outputs the generatedaudio data to the external speaker.

The OSD unit 1248 reproduces the video data inputted from the decoder1247 and displays a video. Further, the OSD unit 1248 may superimpose animage of a GUI such as, for example, a menu, a button, a cursor or thelike on the displayed video.

The control unit 1249 includes a processor such as a CPU or the like anda memory such as a RAM, a ROM and so forth. The memory stores a programto be executed by the CPU, program data and so forth. The program storedin the memory is read into and executed by the CPU, for example, uponactivation of the recording and reproduction apparatus 1240. Byexecuting the program, the CPU controls operation of the recording andreproduction apparatus 1240, for example, in response to an operationsignal inputted from the user interface unit 1250.

The user interface unit 1250 is connected to the control unit 1249. Theuser interface unit 1250 includes, for example, a button and a switchfor allowing a user to operate the recording and reproduction apparatus1240, a reception unit for a remote controlling signal and so forth. Theuser interface unit 1250 detects an operation by the user through thecomponents to generate an operation signal and outputs the generatedoperation signal to the control unit 1249.

In the recording and reproduction apparatus 1240 configured in thismanner, for example, the encoder 1243 may have the function of theencoding apparatus 10 described above. In short, the encoder 1243 mayencode image data by the method described in the foregoing descriptionof the embodiment. By such encoding, the recording and reproductionapparatus 1240 can improve the transmission efficiency and the picturequality.

Further, in the recording and reproduction apparatus 1240 configured inthis manner, for example, the decoder 1247 may have the functions of thedecoding apparatus 20 described hereinabove. In short, the decoder 1247may decode encoded data by the method described in the foregoingdescription of the embodiment. By such decoding the recording andreproduction apparatus 1240 can improve the transmission efficiency andthe picture quality.

Fourth Application Example: Image Pickup Apparatus

FIG. 32 is a view depicting an example of a schematic configuration ofan image pickup apparatus to which the embodiment described hereinaboveis applied.

The image pickup apparatus 1260 picks up an image of an image pickupobject to generate an image and encodes and records image data on arecording medium.

The image pickup apparatus 1260 includes an optical block 1261, an imagepickup unit 1262, a signal processing unit 1263, an image processingunit 1264, a display unit 1265, an external interface (I/F) unit 1266, amemory unit 1267, a media drive 1268, an OSD unit 1269, a control unit1270, a user interface (I/F) unit 1271 and a bus 1272.

The optical block 1261 is connected to the image pickup unit 1262. Theimage pickup unit 1262 is connected to the signal processing unit 1263.The display unit 1265 is connected to the image processing unit 1264.The user interface unit 1271 is connected to the control unit 1270. Thebus 1272 connects the image processing unit 1264, external interfaceunit 1266, memory unit 1267, media drive 1268, OSD unit 1269 and controlunit 1270 to each other.

The optical block 1261 includes a focus lens, a diaphragm mechanism andso forth. The optical block 1261 forms an optical image of an imagepickup object on an image pick plane of the image pickup unit 1262. Theimage pickup unit 1262 includes an image sensor such as a CCD (ChargeCoupled Device) image sensor, a CMOS (Complementary Metal OxideSemiconductor) image sensor or the like and converts an optical imageformed on the image pickup plane into an image signal in the form of anelectric signal by photoelectric conversion. Then, the image pickup unit1262 outputs the image signal to the signal processing unit 1263.

The signal processing unit 1263 performs various camera signal processessuch as knee correction, gamma correction, color correction and so forthfor an image signal inputted from the image pickup unit 1262. The signalprocessing unit 1263 outputs the image data after the camera signalprocesses to the image processing unit 1264.

The image processing unit 1264 encodes the image data inputted from thesignal processing unit 1263 to generate encoded data. Then, the imageprocessing unit 1264 outputs the generated encoded data to the externalinterface unit 1266 or the media drive 1268. Further, the imageprocessing unit 1264 decodes encoded data inputted from the externalinterface section 1266 or the media drive 1268 to generate image data.Then, the image processing unit 1264 outputs the generated image data tothe display unit 1265. Further, the image processing unit 1264 mayoutput image data inputted from the signal processing unit 1263 to thedisplay unit 1265 such that an image is displayed. Further, the imageprocessing unit 1264 may superimpose display data acquired from the OSDunit 1269 on the image to be outputted to the display unit 1265.

The OSD unit 1269 generates an image of a GUI such as, for example, amenu, a button, a cursor or the like and outputs the generated image tothe image processing unit 1264.

The external interface unit 1266 is configured, for example, as a USBinput/output terminal. The external interface unit 1266 connects theimage pickup apparatus 1260 and a printer, for example, upon printing ofan image. Further, as occasion demands, a drive is connected to theexternal interface unit 1266. On the drive, a removable medium such as,for example, a magnetic disk, an optical disk or the like is mounted,and a program read out from the removable medium can be installed intothe image pickup apparatus 1260. Furthermore, the external interfaceunit 1266 may be configured as a network interface connected to anetwork such as a LAN, the internet or the like. In other words, theexternal interface unit 1266 has a role as a transmission unit of theimage pickup apparatus 1260.

The recording medium to be mounted on the media drive 1268 may be areadable and writable arbitrary removable medium such as, for example, amagnetic disk, a magneto-optical disk, an optical disk, a semiconductormemory or the like. Further, a recording medium may be mounted fixedlyon the media drive 1268 such that a non-portable storage unit such as,for example, a built-in type hard disk drive or an SSD (Solid StateDrive) is configured.

The control unit 1270 includes a processor such as a CPU or the like anda memory such as a RAM, a ROM or the like. The memory stores therein aprogram to be executed by the CPU, program data and so forth. Theprogram stored in the memory is read into and executed by the CPU, forexample, upon activation of the image pickup apparatus 1260. Byexecuting the program, the CPU controls operation of the image pickupapparatus 1260, for example, in response to an operation signal inputtedfrom the user interface unit 1271.

The user interface unit 1271 is connected to the control unit 1270. Theuser interface unit 1271 includes, for example, a button, a switch andso forth for allowing the user to operate the image pickup apparatus1260. The user interface unit 1271 detects an operation by the userthrough the components described to generate an operation signal andoutputs the generated operation signal to the control unit 1270.

In the image pickup apparatus 1260 configured in this manner, forexample, the image processing unit 1264 may have the functions of theencoding apparatus 10 described hereinabove. In short, the imageprocessing unit 1264 may encode image data by the method described inthe foregoing description of the embodiments. By such encoding, theimage pickup apparatus 1260 can improve the transmission efficiency andthe picture quality.

Further, in the image pickup apparatus 1260 configured in such a manneras described above, for example, the image processing unit 1264 may havethe functions of the decoding apparatus 20 described hereinabove. Inshort, the image processing unit 1264 may decode encoded data by themethod described in the foregoing description of the embodiment. By suchdecoding, the image pickup apparatus 1260 can improve the transmissionefficiency and the picture quality.

Other Application Examples

It is to be noted that present technology can be applied to such HTTPstreaming as, for example, MPEG DASH or the like in which appropriatedata is selected and used in a unit of a segment from among a pluralityof encoded data prepared in advance and having resolutions or the likedifferent from each other. In short, also it is possible for such aplurality of encoded data as just described to share informationrelating to encoding or decoding.

Further, while the foregoing description is directed to examples of anapparatus, a system and so forth to which the present technology isapplied, the present technology is not limited to this and can becarried out also as any constitution to be incorporated in such anapparatus as described above or an apparatus that configures such asystem as described above, such as, for example, a processor as a systemLSI (Large Scale Integration) or the like, a module that uses aplurality of processors or the like, a unit that uses a plurality ofmodules or the like, a set in which some other function is added to aunit (namely, part of constitutions of an apparatus).

<Video Set>

An example of a case in which the present technology is carried out as aset is described with reference to FIG. 33.

FIG. 33 is a view depicting an example of a schematic configuration of avideo set to which the present technology is carried out as a set.

In recent years, multifunctionalization of electronic apparatus has beenand is proceeding, and in the case where, in development or manufacture,some configuration is carried out as sales, provision or the like, notonly a case in which it is carried out as a constitution having onefunction, but also a case in which a plurality of constitutions havingfunctions associated with each other is combined and carried out as oneset having a plurality of functions are found increasingly.

The video set 1300 depicted in FIG. 33 has such a multifunctionalizedconfiguration and is a combination, with a device having a function orfunctions relating to encoding and/or decoding of an image (one or bothof encoding and decoding), a device having some other function relatingto the function or functions.

As depicted in FIG. 33, the video set 1300 includes a module groupincluding a video module 1311, an external memory 1312, a powermanagement module 1313, a front end module 1314 and so forth, and adevice having relating functions such as a connectivity 1321, a camera1322, a sensor 1323 and so forth.

A module is a part in which several part functions related to each otherare collected such that it has coherent functions. Although a particularphysical configuration is arbitrary, for example, a module isconceivable in which electronic circuit elements having individualfunctions such as a plurality of processors, registers, capacitors andso forth and other devices and so forth are disposed on a wiring boardor the like and integrated. Also, it is conceivable to combine a modulewith another module, a process or the like to form a new module.

In the case of the example of FIG. 33, the video module 1311 is acombination of constitutions having functions relating to imageprocessing and includes an application processor 1331, a video processor1332, a broadband modem 1333 and a RF module 1334.

A processor includes constitutions, which have predetermined functions,integrated on a semiconductor chip by SoC (System on a Chip) and iscalled, for example, system LSI (Large Scale Integration) or the like.The constitutions having the predetermined functions may be logiccircuits (hardware configuration), may be a CPU, a ROM, a RAM and soforth and a program executed using them (software configuration) or maybe a combination of both of them. For example, a processor may include alogic circuit and a CPU, a ROM, a RAM and so forth such that part offunctions is implemented by the logic circuit (hardware configuration)and other functions are implemented by a program (softwareconfiguration) executed by the CPU.

The application processor 1331 of FIG. 33 is a processor that executesan application relating to image processing. The application executed bythe application processor 1331 not only can execute, in order toimplement predetermined functions, arithmetic operation processing butalso can control constitutions inside and outside of the video module1311 such as, for example, the video processor 1332 and so forth ifnecessary.

The video processor 1332 is a processor having a function relating toencoding and/or decoding of an image (one or both of encoding anddecoding).

The broadband modem 1333 performs digital modulation or the like fordata (digital signal) to be transmitted by wired or wireless (or both ofwired and wireless) broadband communication performed through abroadband line such as the Internet, a public telephone network or thelike to convert the data into an analog signal or converts an analogsignal received by such broadband communication to convert the analogsignal into data (digital signal). The broadband modem 1333 processesarbitrary information such as, for example, image data to be processedby the video processor 1332, a stream encoded from image data, anapplication program, setting data or the like.

The RF module 1334 is a module that performs frequency conversion,modulation/demodulation, amplification, filtering and so forth for a RF(Radio Frequency) signal to be transmitted or received through anantenna. For example, the RF module 1334 performs frequency conversionand so forth for a baseband signal generated by the broadband modem 1333to generate a RF signal. Further, for example, the RF module 1334performs frequency conversion and so forth for a RF signal receivedthrough the front end module 1314 to generate a baseband signal.

It is to be noted that, as indicated by a broken line 1341 in FIG. 33,the application processor 1331 and the video processor 1332 may beintegrated so as to be configured as a single processor.

The external memory 1312 is a module provided outside the video module1311 and having a storage device utilized by the video module 1311.Although the storage device of the external memory 1312 may beimplemented by any physical constitution, since generally it isfrequently utilized for storage of a large amount of data such as imagedata of a unit of a frame, it is desirable to implement the storagedevice by a semiconductor device that is comparatively less expensiveand has a large capacity like a DRAM (Dynamic Random Access Memory).

The power management module 1313 manages and controls power supply tothe video module 1311 (constitutions in the video module 1311).

The front end module 1314 is a module that provides a front end function(circuit at a transmission/reception end of the antenna side) to the RFmodule 1334. As depicted in FIG. 33, the front end module 1314 includes,for example, an antenna unit 1351, a filter 1352 and an amplificationunit 1353.

The antenna unit 1351 includes an antenna for transmitting and receivinga wireless signal and peripheral constitutions. The antenna unit 1351transmits a signal supplied from the amplification unit 1353 as awireless signal and supplies a received wireless signal as an electricsignal (RF signal) to the filter 1352. The filter 1352 performs filterprocessing and so forth for a RF signal received through the antennaunit 1351 and supplies the RF signal after the processing to the RFmodule 1334. The amplification unit 1353 amplifies the RF signalsupplied from the RF module 1334 and supplies the amplified RF signal tothe antenna unit 1351.

The connectivity 1321 is a module having functions relating toconnection to the outside. The physical configuration of theconnectivity 1321 is arbitrary. For example, the connectivity 1321includes constitutions having a communication function according to astandard other than a communication standard with which the broadbandmodem 1333 is compatible, external input and output terminals and soforth.

For example, the connectivity 1321 may include a module having acommunication function that complies with a wireless communicationstandard such as Bluetooth (registered trademark), IEEE 802.11 (forexample, Wi-Fi (Wireless fidelity, registered trademark)), NFC (NearField Communication), IrDA (InfraRed Data Association) or the like, anantenna for transmitting and receiving a signal that complies with thestandard, and so forth. Further, for example, the connectivity 1321 mayinclude a module having a communication function that complies with awired communication standard such as USB (Universal Serial Bus), HDMI(registered trademark) (High-Definition Multimedia Interface) or thelike, a terminal that complies with the standard and so forth.Furthermore, for example, the connectivity 1321 may include other data(signal) transmission functions such as analog input and outputterminals and so forth.

It is to be noted that the connectivity 1321 may include a device of atransmission destination of data (signal). For example, the connectivity1321 may include a drive for performing reading out and writing of datafrom and into a recording medium such as a magnetic disk, an opticaldisk, a magneto-optical disk, a semiconductor memory or the like(including not only a drive for a removable medium but also a hard disk,a SSD (Solid State Drive), a NAS (Network Attached Storage) and soforth). Further, the connectivity 1321 may include an outputting deviceof an image or sound (a monitor, a speaker or the like).

The camera 1322 is a module having a function for picking up an image ofan image pickup object to obtain image data of the image pickup object.Image data obtained by image pickup of the camera 1322 is, for example,supplied to and encoded by the video processor 1332.

The sensor 1323 is a module having an arbitrary sensor function such as,for example, a sound sensor, an ultrasonic sensor, an optical sensor, anilluminance sensor, an infrared sensor, an image sensor, a rotationsensor, an angle sensor, an angular velocity sensor, a speed sensor, anacceleration sensor, an inclination sensor, a magnetic identificationsensor, a shock sensor, a temperature sensor and so forth. Data detectedby the sensor 1323 is supplied, for example, to the applicationprocessor 1331 and is utilized by an application or the like.

The constitution described as a module in the foregoing description maybe implemented as a processor, and conversely, the constitutiondescribed as a processor may be implemented as a module.

In the video set 1300 having such a configuration as described above,the present technology can be applied to the video processor 1332 ashereinafter described. Accordingly, the video set 1300 can be carriedout as a set to which the present technology is applied.

<Example of Configuration of Video Processor>

FIG. 34 is a view depicting an example of a schematic configuration ofthe video processor 1332 (FIG. 33) to which the present technology isapplied.

In the case of the example of FIG. 34, the video processor 1332 has afunction for receiving an input of a video signal and an audio signaland encoding them by a predetermined method and a function for decodingencoded video data and audio data and reproducing and outputting a videosignal and an audio signal.

As depicted in FIG. 34, the video processor 1332 includes a video inputprocessing unit 1401, a first image scaling unit 1402, a second imagescaling unit 1403, a video output processing unit 1404, a frame memory1405 and a memory controlling unit 1406. The video processor 1332further includes an encode-decode engine 1407, video ES (ElementaryStream) buffers 1408A and 1408B and audio ES buffers 1409A and 1409B.Furthermore, the video processor 1332 includes an audio encoder 1410, anaudio decoder 1411, a multiplexing unit (MUX (Multiplexer)) 1412, ademultiplexing unit (DMUX (Demultiplexer)) 1413 and a stream buffer1414.

The video input processing unit 1401 acquires a video signal inputted,for example, from the connectivity 1321 (FIG. 33) or the like andconverts the video signal into digital image data. The first imagescaling unit 1402 performs format conversion, a scaling process of animage and so forth for image data. The second image scaling unit 1403performs a scaling process of an image in response to a format at anoutputting designation through the video output processing unit 1404 andperforms format conversion, a scaling process of an image and so forthsimilar to those by the first image scaling unit 1402 for image data.The video output processing unit 1404 performs format conversion,conversion into an analog signal and so forth for image data and outputsthe resulting analog signal as a reproduced video signal, for example,to the connectivity 1321 and so forth.

The frame memory 1405 is a memory for image data shared by the videoinput processing unit 1401, first image scaling unit 1402, second imagescaling unit 1403, video output processing unit 1404 and encode-decodeengine 1407. The frame memory 1405 is implemented as a semiconductormemory such as, for example, a DRAM or the like.

The memory controlling unit 1406 receives a synchronizing signal fromthe encode-decode engine 1407 and controls writing and reading outaccess to the frame memory 1405 in accordance with an access schedule tothe frame memory 1405 written in an access management table 1406A. Theaccess management table 1406A is updated by the memory controlling unit1406 in response to a process executed by the encode-decode engine 1407,first image scaling unit 1402, second image scaling unit 1403 or thelike.

The encode-decode engine 1407 performs an encoding process of image dataand a decoding process of a video stream that is encoded data of imagedata. For example, the encode-decode engine 1407 encodes image data readout from the frame memory 1405 and successively writes the encoded imagedata as a video stream into the video ES buffer 1408A. Further, forexample, the encode-decode engine 1407 successively reads out anddecodes a video stream from the video ES buffer 1408B and successivelywrites the decoded video stream as image data into the frame memory1405. The encode-decode engine 1407 uses the frame memory 1405 as aworking area in such encoding and decoding. Further, the encode-decodeengine 1407 outputs a synchronizing signal to the memory controllingunit 1406 at a timing at which, for example, processing for each macroblock is to be started.

The video ES buffer 1408A buffers a video stream generated by theencode-decode engine 1407 and supplies the buffered video stream to themultiplexing unit (MUX) 1412. The video ES buffer 1408B buffers a videostream supplied from the demultiplexing unit (DMUX) 1413 and suppliesthe buffered video stream to the encode-decode engine 1407.

The audio ES buffer 1409A buffers an audio stream generated by the audioencoder 1410 and supplies the buffered audio stream to the multiplexingunit (MUX) 1412. The audio ES buffer 1409B buffers an audio streamsupplied from the demultiplexing unit (DMUX) 1413 and supplies thebuffered audio stream to the audio decoder 1411.

The audio encoder 1410, for example, digitally converts an audio signalinputted from the connectivity 1321 or the like and encodes the digitalaudio signal by a predetermined method such as, for example, an MPEGaudio method, an AC3 (Audio Code number 3) method or the like. The audioencoder 1410 successively writes an audio stream, which is encoded dataof an audio signal, into the audio ES buffer 1409A. The audio decoder1411 decodes an audio stream supplied from the audio ES buffer 1409B,performs, for example, conversion into an analog signal or the like, andsupplies the resulting analog signal as a reproduced audio signal, forexample, to the connectivity 1321 or the like.

The multiplexing unit (MUX) 1412 multiplexes a video stream and an audiostream. The method of the multiplexing (namely, the format of a bitstream to be generated by the multiplexing) is arbitrary. Further, uponsuch multiplexing, also it is possible for the multiplexing unit (MUX)1412 to add predetermined header information and so forth to the bitstream. In other words, the multiplexing unit (MUX) 1412 can convert theformat of the stream by the multiplexing. For example, the multiplexingunit (MUX) 1412 multiplexes a video stream and an audio stream toconvert the streams into a transport stream that is a bit stream of aformat for transfer. Further, for example, the multiplexing unit (MUX)1412 multiplexes a video stream and an audio stream to convert them intodata of a file format for recording (file data).

The demultiplexing unit (DMUX) 1413 demultiplexes a bit stream, in whicha video stream and an audio stream are multiplexed, by a methodcorresponding to that of the multiplexing by the multiplexing unit (MUX)1412. In short, the demultiplexing unit (DMUX) 1413 extracts a videostream and an audio stream from a bit stream read out from the streambuffer 1414 (separates a video stream and an audio stream from eachother). In short, the demultiplexing unit (DMUX) 1413 can convert theformat of a stream by demultiplexing (reverse conversion to theconversion by the multiplexing unit (MUX) 1412). For example, thedemultiplexing unit (DMUX) 1413 can convert a transport stream supplied,for example, from the connectivity 1321, broadband modem 1333 or thelike into a video stream and an audio stream by acquiring the transportstream through the stream buffer 1414 and demultiplexing the transportstream. Further, for example, the demultiplexing unit (DMUX) 1413 canconvert file data read out from various recording media, for example, bythe connectivity 1321 into a video stream and an audio stream byacquiring the file data through the stream buffer 1414 anddemultiplexing the file data.

The stream buffer 1414 buffers a bit stream. For example, the streambuffer 1414 buffers a transport stream supplied from the multiplexingunit (MUX) 1412 and supplies the buffered transport stream, for example,to the connectivity 1321, broadband modem 1333 and so forth at apredetermined timing or on the basis of a request from the outside orthe like.

Further, the stream buffer 1414 buffers file data supplied from themultiplexing unit (MUX) 1412 and supplies the buffered file data, forexample, to the connectivity 1321 and so forth at a predetermined timingor on the basis of a request from the outside or the like such that thefile data is recorded on various recording media.

Furthermore, the stream buffer 1414 buffers a transport stream acquired,for example, through the connectivity 1321, broadband modem 1333 or thelike and supplies the buffered transport stream to the demultiplexingunit (DMUX) 1413 at a predetermined timing or on the basis of a requestfrom the outside or the like.

Further, the stream buffer 1414 buffers file data read out from variousrecording media, for example, by the connectivity 1321 or the like andsupplies the buffered file data to the demultiplexing unit (DMUX) 1413at a predetermined timing or on the basis of a request from the outsideor the like.

Now, an example of operation of the video processor 1332 having such aconfiguration as described above is described. For example, a videosignal inputted from the connectivity 1321 or the like to the videoprocessor 1332 is converted into digital image data of a predeterminedmethod such as a 4:2:2 Y/Cb/Cr method or the like by the video inputprocessing unit 1401 and is successively written into the frame memory1405. The digital image data are read out into the first image scalingunit 1402 or the second image scaling unit 1403 and subjected to formatconversion to that of a predetermined method such as the 4:2:0 Y/Cb/Crmethod and a scaling process, and then are written into the frame memory1405 again. The image data is encoded by the encode-decode engine 1407and written as a video stream into the video ES buffer 1408A.

Meanwhile, an audio signal inputted from the connectivity 1321 or thelike to the video processor 1332 is encoded by the audio encoder 1410and written as an audio stream into the audio ES buffer 1409A.

The video stream of the video ES buffer 1408A and the audio stream ofthe audio ES buffer 1409A are read out to and multiplexed by themultiplexing unit (MUX) 1412 such that they are converted into atransport stream, file data or the like. The transport stream generatedby the multiplexing unit (MUX) 1412 is buffered by the stream buffer1414 and then is outputted to an external network, for example, throughthe connectivity 1321, broadband modem 1333 or the like. Meanwhile, thefile data generated by the multiplexing unit (MUX) 1412 is buffered bythe stream buffer 1414 and then outputted, for example, to theconnectivity 1321 or the like and then recorded into various recordingmedia.

On the other hand, a transport stream inputted from an external networkto the video processor 1332, for example, through the connectivity 1321,broadband modem 1333 and so forth is buffered by the stream buffer 1414and then demultiplexed by the demultiplexing unit (DMUX) 1413.Meanwhile, file data read out from various recording media, for example,by the connectivity 1321 or the like and inputted to the video processor1332 is buffered by the stream buffer 1414 and then demultiplexed by thedemultiplexing unit (DMUX) 1413. In short, a transport stream or filedata inputted to the video processor 1332 is demultiplexed into a videostream and an audio stream by the demultiplexing unit (DMUX) 1413.

The audio stream is supplied through the audio ES buffer 1409B to anddecoded by the audio decoder 1411 to reproduce an audio signal.Meanwhile, the video stream is successively read out, after written intothe video ES buffer 1408B, and decoded by the encode-decode engine 1407and written into the frame memory 1405. The decoded image data issubjected to a scaling process by the second image scaling unit 1403 andis written into the frame memory 1405. Then, the decoded image data isread out into the video output processing section 1404 and is formatconverted to a predetermined format such as the 4:2:2 Y/Cb/Cr format,whereafter it is converted into an analog signal and a video signal isreproduced and outputted.

In the case where the present technology is to be applied to the videoprocessor 1332 configured in such a manner as described above, thepresent technology according to the embodiment described above may beapplied to the encode-decode engine 1407. In particular, for example,the encode-decode engine 1407 may have the function of the encodingapparatus 10 or the function of the decoding apparatus 20 describedabove or both of them. This makes it possible to obtain advantageouseffects similar to those of the encoding apparatus 10 or the decodingapparatus 20 of the embodiment described hereinabove.

It is to be noted that, in the encode-decode engine 1407, the presenttechnology (namely, the function of the encoding apparatus 10 or thefunction of the decoding apparatus 20 or both of them) may beimplemented by hardware such as logic circuits or the like or may beimplemented by software such as an incorporated program or the like orelse may be implemented by both of them.

<Other Examples of Configuration of Video Processor>

FIG. 35 is a view depicting a different example of a schematicconfiguration of the video processor 1332 to which the presenttechnology is applied.

In the case of the example of FIG. 35, the video processor 1332 has afunction for encoding and decoding video data by a predetermined method.

More particularly, as depicted in FIG. 35, the video processor 1332includes a control unit 1511, a display interface 1512, a display engine1513, an image processing engine 1514 and an internal memory 1515. Thevideo processor 1332 further includes a codec engine 1516, a memoryinterface 1517, a multiplexing and demultiplexing unit (MUX DMUX) 1518,a network interface 1519 and a video interface 1520.

The control unit 1511 controls operation of the processing units in thevideo processor 1332 such as the display interface 1512, display engine1513, image processing engine 1514, codec engine 1516 and so forth.

As depicted in FIG. 35, the control unit 1511 includes, for example, amain CPU 1531, a sub CPU 1532 and a system controller 1533. The main CPU1531 executes a program and for forth for controlling operation of theprocessing units in the video processor 1332. The main CPU 1531generates a control signal in accordance with the program and so forthand supplies the control signal to the processing units (namely,controls operation of the processing units). The sub CPU 1532 plays anauxiliary role of the main CPU 1531. For example, the sub CPU 1532executes a child process, a subroutine or the like of the program or thelike executed by the main CPU 1531. The system controller 1533 controlsoperation of the main CPU 1531 and the sub CPU 1532 such as to designateprograms to be executed by the main CPU 1531 and the sub CPU 1532 or thelike.

The display interface 1512 outputs image data, for example, to theconnectivity 1321 and so forth under the control of the control unit1511. For example, the display interface 1512 converts image data in theform of digital data into an analog signal and outputs the image data asa reproduced video signal, or the image data of digital data as theyare, to a monitor apparatus or the like of the connectivity 1321.

The display engine 1513 performs various conversion processes such asformat conversion, size conversion, color region conversion and so forthfor the image data under the control of the control unit 1511 such thatthe image data satisfies hardware specifications of a monitor apparatusor the like on which an image of the image data is to be displayed.

The image processing engine 1514 carries out predetermined imageprocessing such as, for example, a filtering process and so forth forpicture quality improvement for the image data under the control of thecontrol unit 1511.

The internal memory 1515 is a memory provided in the inside of the videoprocessor 1332 such that it is shared by the display engine 1513, imageprocessing engine 1514 and codec engine 1516. The internal memory 1515is used for transfer of data performed, for example, between the displayengine 1513, image processing engine 1514 and codec engine 1516. Forexample, the internal memory 1515 stores data supplied from the displayengine 1513, image processing engine 1514 or codec engine 1516 andsupplies, as occasion demands (for example, in accordance with arequest), the data to the display engine 1513, image processing engine1514 or codec engine 1516. Although this internal memory 1515 may beimplemented by any storage device, since it is frequently utilized forstorage of a small amount of data such as image data in a unit of ablock, parameters or the like, preferably it is implemented by asemiconductor memory that has a comparatively (for example, incomparison with the external memory 1312) small capacity but is high inresponse speed like, for example, an SRAM (Static Random Access Memory).

The codec engine 1516 performs a process relating to encoding ordecoding of image data. The method for encoding and decoding with whichthe codec engine 1516 is compatible is arbitrary, and the number of suchmethods may be one or a plural number. For example, the codec engine1516 may have codec functions of a plurality of encoding and decodingmethods and perform encoding of image data or decoding of encoded databy a codec function selected from the codec functions.

In the example depicted in FIG. 35, the codec engine 1516 includes, asfunctional blocks for processing relating to the codec, for example, anMPEG-2 Video 1541, an AVC/H.264 1542, an HEVC/H.265 1543, an HEVC/H.265(Scalable) 1544, an HEVC/H.265 (Multi-view) 1545 and an MPEG-DASH 1551.

The MPEG-2 Video 1541 is a functional block that encodes or decodesimage data by the MPEG-2 method. The AVC/H.264 1542 is a functionalblock that encodes or decodes image data by the AVC method. TheHEVC/H.265 1543 is a functional block that encodes or decodes image databy the HEVC method. The HEVC/H.265 (Scalable) 1544 is a functional blockthat scalably encodes or scalably decodes image data by the HEVC method.The HEVC/H.265 (Multi-view) 1545 is a functional block thatmulti-visually encodes or multi-visually decodes image data by the HEVCmethod.

The MPEG-DASH 1551 is a functional block for transmitting and receivingimage data by the MPEG-DASH (MPEG-Dynamic Adaptive Streaming over HTTP)method. MPEG-DASH is a technology for performing streaming of a videousing the HTTP (Hyper Text Transfer Protocol) and has one ofcharacteristics in that appropriate encoded data is selected andtransmitted in a unit of a segment from among a plurality of encodeddata that are prepared in advance and are different from each other inresolution and so forth. The MPEG-DASH 1551 performs generation of astream that complies with the standard, transmission control of thestream and so forth, and for encoding and decoding of image data, theMPEG-2 Video 1541 to HEVC/H.265 (Multi-view) 1545 described above areutilized.

The memory interface 1517 is an interface for the external memory 1312.Data supplied from the image processing engine 1514 or the codec engine1516 is supplied to the external memory 1312 through the memoryinterface 1517. Meanwhile, data read out from the external memory 1312is supplied to the video processor 1332 (image processing engine 1514 orcodec engine 1516) through the memory interface 1517.

The multiplexing and demultiplexing unit (MUX DMUX) 1518 performsmultiplexing and demultiplexing of various data relating to an imagesuch as a bit stream of encoded data, image data, a video signal and soforth. The method for the multiplexing and demultiplexing is arbitrary.For example, upon multiplexing, the multiplexing and demultiplexing unit(MUX DMUX) 1518 not only can collect a plurality of data into one databut also can add predetermined header information and so forth to thedata. Further, upon demultiplexing, the multiplexing and demultiplexingunit (MUX DMUX) 1518 not only can divide one data into a plurality ofdata but also can add predetermined header information and so forth tothe divisional data. In short, the multiplexing and demultiplexing unit(MUX DMUX) 1518 can convert the data format by multiplexing anddemultiplexing. For example, the multiplexing and demultiplexing unit(MUX DMUX) 1518 can multiplex bit streams to convert them into atransport stream that is a bit stream of a format for transfer or dataof a file format for recording (file data). Naturally, the multiplexingand demultiplexing unit (MUX DMUX) 1518 can perform inverse conversionby demultiplexing.

The network interface 1519 is an interface, for example, for thebroadband model 1333, connectivity 1321 and so forth. The videointerface 1520 is an interface, for example, for the connectivity 1321,camera 1322 and so forth.

Now, an example of operation of such a video processor 1332 as describedabove is described. For example, if a transport stream is received froman external network through the connectivity 1321, broadband modem 1333or the like, then the transport stream is supplied through the networkinterface 1519 to and demultiplexed by the multiplexing anddemultiplexing unit (MUX DMUX) 1518 and then is decoded by the codecengine 1516. Image data obtained by decoding of the codec engine 1516 issubjected, for example, to predetermined image processing by the imageprocessing engine 1514 and is supplied, for example, to the connectivity1321 or the like through the display interface 1512 such that an imagethereof is displayed on the monitor. Meanwhile, for example, image dataobtained by decoding of the codec engine 1516 is re-encoded by the codecengine 1516 and multiplexed by the multiplexing and demultiplexing unit(MUX DMUX) 1518 such that it is converted into file data. The file datais outputted to the connectivity 1321 or the like through the videointerface 1520 and is recorded on various recording media.

Furthermore, file data of encoded data, which are encoded image data,read out from a recording medium not depicted, for example, by theconnectivity 1321 or the like are supplied through the video interface1520 to and demultiplexed by the multiplexing and demultiplexing unit(MUX DMUX) 1518, whereafter they are decoded by the codec engine 1516.Image data obtained by the decoding of the codec engine 1516 aresubjected to predetermined image processing by the image processingengine 1514 and further to predetermined conversion by the displayengine 1513, whereafter they are supplied through the display interface1512, for example, to the connectivity 1321 or the like such that animage thereof is displayed on the monitor. Meanwhile, for example, imagedata obtained by decoding of the codec engine 1516 are re-encoded by thecodec engine 1516 and multiplexed by the multiplexing and demultiplexingunit (MUX DMUX) 1518 such that they are converted into a transportstream. The transport stream is supplied, for example, to theconnectivity 1321, broadband modem 1333 and so forth through the networkinterface 1519 and transmitted to a different apparatus not depicted.

It is to be noted that transfer of image data or other data between theprocessing units in the video processor 1332 is performed utilizing, forexample, the internal memory 1515 or the external memory 1312. Further,the power management module 1313 controls power supply, for example, tothe control unit 1511.

In the case where the present technology is applied to the videoprocessor 1332 configured in such a manner as described above, thepresent technology according to the embodiment described hereinabove maybe applied to the codec engine 1516. In short, for example, the codecengine 1516 may have the function of the encoding apparatus 10 or thefunction of the decoding apparatus 20 described above or both of them.By this, advantageous effects similar to those of the encoding apparatus10 and the decoding apparatus 20 described hereinabove can be achieved.

It is to be noted that, in the codec engine 1516, the present technology(namely, the functions of the encoding apparatus 10 and the decodingapparatus 20) may be implemented by hardware such as logic circuits andso forth, may be implemented by software such as an embedded program orthe like or may be implemented by both of them.

While two examples of the configuration of the video processor 1332 areexemplified above, the configuration of the video processor 1332 isarbitrary and may be any other than the two examples described above.Further, although this video processor 1332 may be configured as asingle semiconductor chip, it may otherwise be configured as a pluralityof semiconductor chips. For example, the video processor 1332 may be,for example, a three-dimensional layered LSI in which a plurality ofsemiconductors are stacked. Further, the video processor 1332 may beimplemented by a plurality of LSIs.

<Example of Application to Apparatus>

The video set 1300 can be incorporated into various apparatus thatprocess image data. For example, the video set 1300 can be incorporatedinto the television apparatus 1200 (FIG. 29), portable telephone set1220 (FIG. 30), recording and reproduction apparatus 1240 (FIG. 31),image pickup apparatus 1260 (FIG. 32) and so forth. By incorporating thevideo set 1300, the apparatus can achieve advantageous effects similarto those of the encoding apparatus 10 or the decoding apparatus 20described hereinabove.

It is to be noted that even part of the constitutions of the video set1300 described above can be carried out as the configuration to whichthe present technology is applied if it includes the video processor1332. For example, only the video processor 1332 can be carried out as avideo processor to which the present technology is applied. Further, forexample, the processor indicated by the broke line 1341, the videomodule 1311 or the like can be carried out as a processor, a module orthe like to which the present technology is applied as describedhereinabove. Furthermore, for example, the video module 1311, externalmemory 1312, power management module 1313 and front end module 1314 canbe combined such that they are carried out as a video unit 1361 to whichthe present technology is applied. In the case of any configuration,advantageous effects similar to those of the encoding apparatus 10 ordecoding apparatus 20 described above can be achieved.

In short, any configuration can be incorporated into various apparatusthat process image data similarly as in the case of the video set 1300if the configuration includes the video processor 1332. For example, thevideo processor 1332, processor indicated by the broken line 1341, videomodule 1311, video unit 1361 can be incorporated into the televisionapparatus 1200 (FIG. 29), portable telephone set 1220 (FIG. 30),recording and reproduction apparatus 1240 (FIG. 31), image pickupapparatus 1260 (FIG. 32) and so forth. Thus, by incorporating any of theconfigurations to which the present technology is applied, the apparatuscan achieve advantageous effects similar to those of the encodingapparatus 10 or the decoding apparatus 20 described hereinabovesimilarly as in the case of the video set 1300.

It is to be noted that the embodiment of the present technology is notlimited to the embodiment described hereinabove but can be altered invarious manners without departing from the subject matter of the presenttechnology.

For example, in the present specification, the term system signifies aset of plural constitutions (apparatus, modules (parts) and so forth)and does not matter whether or not all constitutions are placed in asame housing. Accordingly, both of a plurality of apparatus that areaccommodated in separate housings and are connected to each other by anetwork and one apparatus in which a plurality of modules areaccommodated in one housing are systems.

Further, for example, a constitution described as one apparatus (or oneprocessing unit) may be divided and configured as a plurality ofapparatus (or processing units). Conversely, constitutions described asa plurality of apparatus (or processing units) in the foregoingdescription may be collected such that they are configured as oneapparatus (processing unit). Further, a constitution other than thosemay naturally be added to the configuration of each apparatus (or eachprocessing unit). Furthermore, if a constitution or operation as anentire system is substantially same, then part of constitutions of acertain apparatus (or a certain processing unit) may be included inconstitutions of a different apparatus (or a difference processingunit).

Further, for example, the present technology can assume a configurationfor cloud computing in which one function is shared and processed incooperation by a plurality of apparatus through a network.

Further, for example, the program described hereinabove can be executedby an arbitrary apparatus. In this case, the apparatus may be configuredsuch that it has necessary functions (functional blocks and so forth)and can acquire necessary information.

Further, for example, the steps described in connection with the flowcharts described hereinabove can be executed by one apparatus andfurther can be shared and executed by a plurality of apparatus.Furthermore, in the case where a plurality of processes are included inone step, the plurality of processes included in the one step can beexecuted by one apparatus and also can be shared and executed by aplurality of apparatus.

It is to be noted that the program to be executed by the computer may beof the type by which the processes at steps by which the program isdescribed are executed in a time series in the order as described in thepresent specification or of the type by which the processes are executedin parallel or executed individually at necessary timings such as whenthe process is called. In short, the processes at the steps may beexecuted in an order different from the order described hereinaboveunless inconsistency occurs. Furthermore, the processes at the steps bywhich the program is executed may be executed in parallel to processesof a different program or may be executed in combination with processesof a different apparatus.

It is to be noted that the plurality of present technologies describedin the present specification can individually be carried out solely andindependently of each other unless inconsistency occurs. Naturally, alsoit is possible to carry out an arbitrary plurality of presenttechnologies in combination. For example, also it is possible to carryout the present technology described in the description of anyembodiment in combination with the present technology described in thedescription of a different embodiment. Also, it is possible to carry outan arbitrary one of the present technologies described hereinabove incombination with a different technology that is not describedhereinabove.

It is to be noted that the advantageous effects described in the presentspecification are exemplary to the last and are not restrictive, andother advantageous effects may be applicable.

It is to be noted that the present technology can take the followingconfigurations.

<1>

An encoding apparatus, including:

an encoding unit configured to encode an input image by a non-reversibleencoding method;

a database in which a plurality of texture components are registered;and

a transmission unit configured to transmit identification informationfor identifying match components that are, from among the plurality oftexture components registered in the database, the texture componentsthat match with the input image and encoded data obtained by encodingthe input image.

<2>

The encoding apparatus according to <1>, in which

bases of the texture components obtained by converting the texturecomponents each into a basis are registered in the database.

<3>

The encoding apparatus according to <2>, further including:

a separation unit configured to separate a low frequency component ofthe input image from the input image;

a basis synthesis unit configured to generate, in regard to each of theplurality of texture components registered in the database, arestoration component that restores a texture component of the inputimage by basis synthesis in which the low frequency component of theinput image and the basis of the texture component are used; anda match component determination unit configured to determine, as thematch component, the restoration component whose error with respect tothe input image is in the minimum from among the restoration componentsindividually generated in regard to the plurality of texture componentsregistered in the database.

<4>

The encoding apparatus according to <2>, further including:

a decoding unit configured to decode the encoded data into a decodedimage;

a basis synthesis unit configured to generate, in regard to each of theplurality of texture components registered in the database, arestoration component that restores the texture component of the inputimage by basis synthesis in which the decoded image and the basis of thetexture component are used; anda match component determination unit configured to determine, as thematch component, the restoration component whose error with respect tothe input image is in the minimum from among the restoration componentsindividually generated in regard to the plurality of texture componentsregistered in the database.

<5>

The encoding apparatus according to any one of <2> to <4>, furtherincluding:

a data transmission unit configured to transmit data of the database inresponse to a request from a decoding apparatus that decodes the encodeddata.

<6>

The encoding apparatus according to any one of <2> to <4>, furtherincluding:

an updating unit configured to acquire data from a server and update thedatabase.

<7>

The encoding apparatus according to any one of <2> to <6>, furtherincluding:

a registration unit configured to register a basis of the texturecomponent into the database.

<8>

The encoding apparatus according to <7>, in which the registration unit

temporarily registers the basis of the texture component of the inputimage as a basis of a new texture component into the database, and

definitively registers the basis of the new texture component where agiven condition is satisfied.

<9>

The encoding apparatus according to <8>, in which the registration unitdefinitively registers the basis of the new texture component into thedatabase taking it as the given condition that:

an S/N (Signal to Noise ratio) of the match component where the basis ofthe new texture component is registered in the database with respect tothe input image is superior by a fixed value or more to an S/N of thematch component where the basis of the new texture component is notregistered in the database with respect to the input image; or that anRD (Rate-Distortion) curve where the basis of the new texture componentis registered in the database is superior by a fixed value or more to anRD curve where the basis of the new texture components is not registeredin the database.

<10>

The encoding apparatus according to <8> or <9>, in which,

where an error of the match component where the basis of the new texturecomponent is not registered in the database with respect to the inputimage is equal to or greater than a threshold value, the registrationunit temporarily registers a basis of the texture component of the inputimage as a basis of the new texture component into the database.

<11>

The encoding apparatus according to any one of <1> to <10>, in which

the encoding unit encodes a difference between the input image and thematch component.

<12>

An encoding method, including:

encoding an input image by a non-reversible encoding method; and

transmitting identification information for identifying a matchcomponent that is a texture component that matches with the input imagefrom among a plurality of texture components registered in a database inwhich the plurality of texture components are registered and encodeddata obtained by encoding the input image.

<13>

A decoding apparatus, including:

a reception unit configured to receive encoded data obtained by encodingan input image by a non-reversible encoding method and identificationinformation for identifying a match component that is a texturecomponent that matches with the input image;

a decoding unit configured to decode the encoded data into a decodedimage;

a database in which a plurality of texture components are registered;and

a synthesis unit configured to synthesize the texture component as thematch component identified by the identification information from amongthe plurality of texture components registered in the database and thedecoded image.

<14>

The decoding apparatus according to <13>, in which

bases of the texture components obtained by converting the texturecomponents are registered in the database.

<15>

The decoding apparatus according to <14>, further including:

a basis synthesis unit configured to generate a restoration componentthat restores the match component by basis synthesis in which thedecoded image or a low frequency component of the decoded image and abasis of the match component are used; and in which the synthesis unitsynthesizes the restoration component and the decoded image.

<16>

The decoding apparatus according to <14> or <15>, further including:

an updating unit configured to request an encoding apparatus, whichencodes the input image, and acquire data to be registered into thedatabase to update the database.

<17>

The decoding apparatus according to <14> or <15>, further including:

an updating unit configured to acquire data from a server to update thedatabase.

<18>

A decoding method, including:

receiving encoded data obtained by encoding an input image by anon-reversible encoding method and identification information foridentifying a match component that is a texture component that matcheswith the input image;

decoding the encoded data into a decoded image; and

synthesizing the texture component as the match component identified bythe identification information from among a plurality of texturecomponents registered in a database in which the plurality of texturecomponents are registered and the decoded image.

REFERENCE SIGNS LIST

10 Encoding apparatus, 11 Texture component extraction unit, 12 Removalunit, 13 Encoding unit, 20 Decoding apparatus, 21 Decoding unit, 11Texture component restoration unit, 23 Synthesis unit, 30 Encodingapparatus, 31 Texture DB, 32 Texture component acquisition unit, 33Removal unit, 34 Encoding unit, 35 Transmission unit, 40 Decodingapparatus, 41 Reception unit, 42 Decoding unit, 43 Texture DB, 44Texture component acquisition unit, 45 Synthesis unit, 50 Encodingapparatus, 51 Texture DB, 52 Separation unit, 53 Basis synthesis unit,54 Match component determination unit, 55 Removal unit, 56 Encodingunit, 57 Transmission unit, 60 Decoding apparatus, 61 Reception unit, 62Decoding unit, 63 Texture DB, 64 Basis synthesis unit, 65 Separationunit, 66 Synthesis unit, 81 Decoding unit, 101 Data transmission unit,111, 121, 131 Updating unit, 141, 141 Server, 151 Registration unit, 161Basis learning unit, 162 Registration decision unit.

What is claimed is:
 1. An encoding apparatus, comprising: an encodingunit configured to encode an input image by a non-reversible encodingmethod; a database configured to register a plurality of texturecomponents; a separation unit configured to separate a low frequencycomponent of the input image from the input image; a basis synthesisunit configured to generate a restoration component for each texturecomponent of the plurality of texture components registered in thedatabase, wherein the restoration component restores a texture componentof the input image, and the restoration component is generated based ona basis synthesis in which the low frequency component of the inputimage and a basis of a texture component of the plurality of texturecomponents are used; a match component determination unit configured todetermine, as a match component, the restoration component whose errorwith respect to the input image is a minimum from among a plurality ofrestoration components generated for the plurality of texture componentsregistered in the database; and a transmission unit configured totransmit identification information and encoded data, wherein theidentification information is for identification of the match componentfrom among the plurality of texture components registered in thedatabase, the match component is a texture component that matches withthe input image, and the encoded data is obtained by encode of the inputimage.
 2. An encoding apparatus, comprising: an encoding unit configuredto encode an input image by a non-reversible encoding method; a databaseconfigured to: register a plurality of texture components; and registera plurality of bases of the plurality of texture components, wherein theplurality of bases is obtained based on conversion of each texturecomponent of the plurality of texture components into a basis; aregistration unit configured to: register a basis of a texture componentof the input image into the database; temporarily register the basis ofthe texture component of the input image as a basis of a new texturecomponent into the database; and definitively register the basis of thenew texture component based on satisfaction of a given condition; and atransmission unit configured to transmit identification information andencoded data, wherein the identification information is foridentification of a match component from among the plurality of texturecomponents registered in the database, the match component is a texturecomponent that matches with the input image, and the encoded data isobtained by encode of the input image.
 3. The encoding apparatusaccording to claim 2, wherein the given condition is one of: an S/N(Signal to Noise ratio) of the match component where the basis of thenew texture component is registered in the database with respect to theinput image is superior by a first fixed value or more to an S/N of thematch component where the basis of the new texture component is notregistered in the database with respect to the input image; or an RD(Rate-Distortion) curve where the basis of the new texture component isregistered in the database is superior by a second fixed value or moreto an RD curve where the basis of the new texture component is notregistered in the database.
 4. The encoding apparatus according to claim2, wherein, based on an error of the match component where the basis ofthe new texture component is not registered in the database with respectto the input image that is equal to or greater than a threshold value,the registration unit is further configured to temporarily register thebasis of the texture component of the input image as the basis of thenew texture component into the database.
 5. An encoding method,comprising: encoding an input image by a non-reversible encoding method;registering a plurality of texture components in a database; separatinga low frequency component of the input image from the input image;generating a restoration component for each texture component of theplurality of texture components registered in the database, wherein therestoration component restores a texture component of the input image,and the restoration component is generated based on a basis synthesis inwhich the low frequency component of the input image and a basis of atexture component of the plurality of texture components are used;determining, as a match component, the restoration component whose errorwith respect to the input image is a minimum from among a plurality ofrestoration components generated for the plurality of texture componentsregistered in the database; and transmitting identification informationand encoded data, wherein the identification information is foridentification of the match component from among the plurality oftexture components registered in the database, the match component is atexture component that matches with the input image, and the encodeddata is obtained by encoding the input image.
 6. An encoding method,comprising: encoding an input image by a non-reversible encoding method;registering a plurality of texture components; registering a pluralityof bases of the plurality of texture components, wherein the pluralityof bases is obtained based on conversion of each texture component ofthe plurality of texture components into a basis; registering a basis ofa texture component of the input image into a database; temporarilyregistering the basis of the texture component of the input image as abasis of a new texture component into the database; definitivelyregistering the basis of the new texture component based on satisfactionof a given condition; and transmitting identification information andencoded data, wherein the identification information is foridentification of a match component from among the plurality of texturecomponents registered in the database, the match component is a texturecomponent that matches with the input image, and the encoded data isobtained by encoding the input image.
 7. A decoding apparatus,comprising: a reception unit configured to receive encoded data andidentification information from an encoding apparatus, wherein theencoded data is obtained by encode of an input image by a non-reversibleencoding method, the identification information is for identification ofa match component that is a texture component that matches with theinput image, the match component corresponds to a restoration componentwhose error with respect to the input image is a minimum from among aplurality of restoration components generated for a plurality of texturecomponents, the plurality of restoration components is generated by theencoding apparatus based on a first basis synthesis in which a firstdecoded image and a basis of a texture component of the plurality oftexture components are used, the encoded data is decoded into the firstdecoded image by the encoding apparatus, and the restoration componentrestores a texture component of the input image; a decoding unitconfigured to decode the encoded data into a second decoded image; adatabase configured to: register the plurality of texture components;and register a plurality of bases of the plurality of texturecomponents, wherein the plurality of bases is obtained based onconversion of each texture component of the plurality of texturecomponents into a basis; a basis synthesis unit configured to generate arestored component based on a second basis synthesis in which the seconddecoded image or a low frequency component of the second decoded imageand a basis of the match component are used; and a synthesis unitconfigured to synthesize the restored component, the second decodedimage and the match component, wherein the match component is identifiedfrom among the plurality of texture components registered in thedatabase, and the match component is identified by the identificationinformation.
 8. The decoding apparatus according to claim 7, furthercomprising an updating unit configured to: request the encodingapparatus which encodes the input image; and acquire data to beregistered into the database to update the database.
 9. The decodingapparatus according to claim 7, further comprising an updating unitconfigured to acquire data from a server to update the database.
 10. Adecoding method, comprising: receiving encoded data and identificationinformation from an encoding apparatus, wherein the encoded data isobtained by encoding an input image by a non-reversible encoding method,the identification information is for identification of a matchcomponent that is a texture component that matches with the input image,the match component corresponds to a restoration component whose errorwith respect to the input image is a minimum from among a plurality ofrestoration components generated for a plurality of texture components,the plurality of restoration components is generated by the encodingapparatus based on a first basis synthesis in which a first decodedimage and a basis of a texture component of the plurality of texturecomponents are used, and the encoded data is decoded into the firstdecoded image by the encoding apparatus; restoring a texture componentof the input image; decoding the encoded data into a second decodedimage; registering the plurality of texture components in a database;registering a plurality of bases of the plurality of texture componentsin the database, wherein the plurality of bases is obtained based onconversion of each texture component of the plurality of texturecomponents into a basis; generating a restored component based on asecond basis synthesis in which the second decoded image or a lowfrequency component of the second decoded image and a basis of the matchcomponent are used; and synthesizing the restored component, the seconddecoded image and the match component, wherein the match component isidentified from among the plurality of texture components registered inthe database, and the match component is identified by theidentification information.